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PREFACE 



Who should read this Guide 

This Guide issues from our own year- long work in developing 
a mastery learning system in the money management area of bank 
checking for dlder teens and adult basic education students who 
rtfad two or more years below grade level. Through trial and 
error, we developed a very effective, in some ways "ideal 11 , 
mastery learning system that we are now publishing in kit form. 
We are practitioners, not researchers or academics, and we 
realize that there is no reason why other practitioners in various 
kinds of school systems and institutions cannot create their wn 
mastery learning systems. We learned a lot in builcy^tg our system 
and now we want t£ give you some "inside tips" and general sug- 
gestions to help you build your own systems, or just improve some 
of your own materials. 

Our kit turned out to be a little more "ideal" than originally 
intended for the small subject area of bank checking. We created 
a lot of new teaching materials that go beyond the basics of 
mastery learning. We're glad now'that we did, but it did require 
extra efforts and costs that you may not be willing to incur. 
Nevertheless, our experiences will still be highly relevant to your 
own case no matter how modest your plans are for developing ma- 
terials or activities for reteaching students who do not pass this 
or that lesson 1 s toastery test. It is essential to the concept 
of mastery learning that students have a chance to relearn material 
they could not master the first time around, so you will want to 
develop at least some reteaching materials to go along with the 
mastery tests you will develop. This Guide will give you all the 
information you need to develop whatever mastery tests and re- 
teaching materials you have in mind. 



Origins of this Guide 

Laubach Literacy International (LLI) is a private, non-profit, 
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educational agency whose mission is to enable literacy-deficient 
adults and teenagers the world over to learn basic skills. LLI 
supports volunteer-based tutoring programs in the U.S. and abroad. 
It also publishes teaching and learning materials not only for 
students and tutors in those programs but also for students and 
teachers in remedial and special education high school classes, 
in Adult Basic Education programs, and in other basic skills pro- 
grams in a variety of institutions and agencies. 

In the Summer of 1980,; LLI applied to the Office of Consumer 
Education, U.S. Department of Education, foxj£& grant-in-aid to 
develop and publish mastery learning materials in the consumer 
education area of money management, particularly bank checking. 
The grarjt was awarded and the project begun in October, 1980. 
One year later, the project produced, field-tested, and published 
a mastery learning kit entitled Mastering Checking ., 

This kit is close to being " ideal" because it includes three 
correlated student texts (readability: 3.5 grade level, Gunning): 

* Understanding Checking (64 pages) 

* Checking Illustrated (96 pages) 

* Checking Practice (24 pages). 

The kit also includes: 

* Practice forms (sample blank checks, deposit slips, etc.), 

* Answer keys (one set for teacher use, another for student 
self -correction) . 

* Teacher's Guide . 

Teachers who use this kit are supposed to select either 
Understanding Checking or Checking Illustrated to act as the 
"Main Text ,f which their students will primarily learn from. Stu- 
dents take a mastery test at the end of each lesson. (The lessons 
in those two alternate Main Texts are correlated to each other 
and teach the same topics but use different teaching methodologies 
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and approaches.) Students who do not pass a mastery -test are 
'called "non-masters"; they go to the correlated lesson in* the 

"Alternate Text" to relearn the lesson f s material. Masters go 
'on to the next lesson in the Main Text, 

The lesson non-masters study in the Alternate Text is corre- 
lated to the lesson in the Main Text which they didn f t raasterV * 
This correlated lesson reteaches the same content using a different 
approach. The Alternate Text is Checking Illustrate^ if the teach- <• 
er selected Understanding Checking to act as the Main Jext. The 
Alternate Text is Checking Practice if the teacher selected Checking 
Illustrated to act as the Main Text. 

Retaught non-masters take a second mastery test at the end of 
the reteaching lesson. If, they pass, they are genuine masters of 
the lesson and go back to the Main Text to start the next lesson. 
If they still did not pass this second mastery test, they get extra 
help from the teacher. 

The chief innovations in our kit are that: 

* Both Understanding Checking and Checking Illustrated can stand 
on their own and act as the Main Text, and 

* Checking Illustrated can act either as the Main Text or as the 
? 

Alternate Text and reteaching complement to Understanding Check - 
ing when the latter acts as the Main Text. 



The reason why the kit is designed to provide two alternatives 
instead of just one is that Understanding Checking is a better 
Main Text for students who are not reluctant readers (but still 
read well below grade level) and who learn better- when the text 
explains "why" as well as describes "how to". On the other hand, 
Checking Illustrated is a better Main Text for students who are 
reluctant readers and are more motivated by fewer words on the 
page and more illustrations and cartoons; this text stresses "how 
to" and explains things less thoroughly. Teachers < n choose 



which of these two alternatives is better for their own classes. 



The team which developed these materials fields tested them 
in seventeen classrooms in central New York State. Generally, 
the kit worked very well. The field test proved to us that 
adult and older teens who are "high risk" slower learners or are ^ 
educable mentally handicapped students do strongly prefer 
mastery learning to other learning methods. However, their 
teachers had trouble using the kit both because it is a little 
more complicated than what they f re used to, and because the kit 
itself takes on more' of the responsibility for the primary ' 
teaching; the teacher manages the kit's use by students and pro- 
vides individual help when students get in trouble and not all 
teachers appreciated this change in their roles in the classroom. 

The team included professional practitioners, writers, 
editors, and test developers. But none in the team had any prior 
direct experience with mastery learning. We feel that other 
groups in the U.S. who are thinking about developing their own 
materials will appreciate reading about the lessons we ourselves 
have leaitied in oi^r year- long development effort. Perhaps some of 
you will not only recognize how powerful* mastery learning is but 
will also realize that it won't be that hard for you to develop 
your own mastery learning system. We have put together this 
Guide in hopes that you will be able to profit from our mistakes 
and emulate our achievements. 
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PR EVIEW 



This Guide is ±ct up. so you can either read it straight 
through as a continuous narrative or use it as a col- 
lection cf separate and independent resource topics you 
can look up when you need to. For this latter purpose , 
rely on the detailed Table Of Contents to act as an 
index of resource sections. 



PART ONE: What is Mastery Learning, and Why Is It Superior ? 

Highlights of how and why mastery learning Works. Should 
give you enough for you to decide whether it Would be worth 
your while to further explore the possibility of using 
mastery learning to improve the instructional prowess of 
your own curriculum. 

PART TWO: Developing a Mastery Learning System: Key Decisions 
and Actions 

Tells you how to create your own mastery learning materials. 
Describes a step-by-step procedure for planning and carrying 
out your development project based on our own experience in 
developing a mastery learning kit.* 



In. our project 9 we did develop mastery tests for the lessons in a 
text on bank checking Which we had already published. But ve also developed 
two new student texts, correlated to the lessons in that published text, 
which Vers much nore thorough, detailed, and effective than the original 
published text. 

At first, these two new texts were supposed to merely give students 
two alternative methods for relearning material in the original published 
text which they had not mastered. However, both of these alternative teach- 
ing methods turned out to be so vastly superior to the method used in the 
original published text that we decided to make the two new texts stand on 
their own and act as the core of our present kit (the original published 
text was dropped from the kit). 

So, our project may have been more ambitious than what you have in 
mind for your own project. If so, don't worry - you will still be able to 
easily adapt our suggestions in this Guide to your own case. On the othe* 
hand, if you do start out with an initially more modest project, there is 
still the possibility that you might face the situation we found ourselves 
faclAg: ^whether to stick with your already existing teaching materials that 
you discover are very deficient or to develop new and vastly superior teach- 
ing materials. If this does happen, the Guide below will help you through 
your difficulties. - 



PART TWO (continued) 

STAGE ONE: PRE-PLANNING DECISIONS 

Checklist of things to consider when deciding whether you 
need and can afford to launch a mastery learning develop- 
ment projec . 

STAGE TWO: Planning and Preparation 

Checklist of steps to take in planning, staffing, and 
financing the mastery learning development project, and 
in preparing your organization to buy into it, 

STAGE THREE: Prepare Frameworks and Models 

Checklist of steps to take in starting the project; further 
planning, working with target teachers , § creating model tests 
and model teaching materials. How to develop mastery tests. 
Discussion of issues and details based on our experiences 
in this stage. 

STAGE FOUR: First Drafts and Pilot-Test Preparations 

Checklist of steps to take. Further discussion of how to 
write the teaching and reteaching materials. Instructional 
methods we use \ Technical issues: graphics, formats, etc. 

STAGE FIVE: Conduct the Pilot-Test of the First Draft 

Checklist of steps to take. Nine points to keep in mind 
during the pilot test. Comments from our observation 
team leader. Issues to be checked out by observers of 
pilot test classrooms. 

STAGE SIX: Revise the Pilot-tested First Draft 

Checklist of steps to take. Brief account of some of 
the revisions we had to make in our first draft after 
we pilot tested and field tested it. 
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STAGES SEVEN-TEN: Field-Test. Final Revisions, Production, and 

Publication and Marketing of the Materials 

These stages go beyond the scope of this Guide because 
they have nothing specifically to do with either mastery 
learning per se or development per se. They also require 
special technical expertise best provided by professional 
evaluators and publishers. Detailed descriptions of these 
stages have therefore been omitted. 
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PART ONE: WHAT IS MASTERY LEARNING, AND WHY IS IT SUPERIOR ? 

Basic mastery learni ng 

MAstery learning is a special way of teaching that en- 
ables nearly all students to master what they study. Virtually 
all basic education curricula can be converted into mastery 
learning systems . 

In mastery learning, a student works through a lesson 
and then takes a mastery test* If the student passes the 
mastery test, then he or she can go on to the next lesson. If 
the student doesn't pass the test, he or ■ she gets a second 
chance and works through some remedial instruction on that 
lessen. Then the student takes a second mastery test on the 
same material. A passing score means the student is ready to 
go on. If the student fails to pass this second time, he or 
she gets a third chance and receives individualized help from 
the teacher in order to reach mastery. 

That 1 s really all there is to it. The point is that 
students don f t tackle new lessons before they've mastered ear- 
lier lessons but they always get another chance to succeed. 

What: is ENHANCED mastery learning ? 

In basic mastery learning, the remedial or reteaching 
instruction that nonmasters work through can take the form of 
just restudying the same lesson. Or it may consist in the 
teacher working with the nonmaster or referring the nonmaster 
to other available and related materials. 

In enhanced mastery learning, specially rrepaiod reteach- 
ing materials have been developed in advance. They are spe- 
cifically for reteaching the same content taugnt in the original 
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main lesson. And they deliberately reteach that content in a 
different (usually simpler and more concrete) way. Enhanced 
mastery learning is better than basic mastery learning, but it 
is also more expensive and difficult to develop. 



HOW DOES IT WORK? 

Independent ^earning 

All students work on their own 
through a lesson in the Main Text: 




Mastery Testing 
When he/she is ready, each student 
takes a Mastery lest on that lesson, 
teacher corrects and scores it: 



New learning approach for 
Reteaching nonmasters 
Masters go on to the next lesson in 
the Main Text. 

Nonmasters go to the lesson's corre- 
lated reteaching lesson in the Re- 
teaching Text, take a second Mastery 
lest. When they master the material, 
they go on to the next lesson in the 
Main Text. 





A more detailed picture of the enhanced mastery learning 
process that we use in our kit is displayed on the next two 
pages. Please refer back to these diagrams as you read through 
the rest of this guide. 
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For classes In which Individuals go at thair own pace^the process will look like this: 
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Review 
Tutoring 
Remedial 
activities 
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mastery 
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nonmasters? 
Decision: 

Go to next section 
or keep trying in 
this section* 



For classes that proceed together the process will look like this: 
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What does the research say ? 

A considerable amount of educational and psychological 
research has proven that 

1. Frequent testing with immediate feedback and re- 
ceaching of unmastered material in early lessons 
tends to strengthen the desire to learn and build 
self-confidence. 

2. Thorough learning (■ mastery) of earlier material 
tends to facilitate and quicken the learning of 
later material; although sometimes slower and more 
time-consuming than conventional methods at the out- 
let of a course, mastery learning is often less time- 
consuming in the long-run because later learning is 
faster. 

* 

In all mastery learning, nearly all students succeed , - 

The basic assumption of mastery learning is that most 
students can master what is being taught, In the typical non- 
mastery classroom, only the best students are expected to 
achieve mastery. 



In the usual American school, only those students who can 
achieve high scores in a certain amount of time are judged to 
be masters. The underlying assumption is that a "good" student 
is one who learns more quickly than others. Most of the psycho- 
logical rewards go to the quick learners, and most of the psycho- 
logical punishment goes to the slow ones. 

Mastery learning asks, "Does it really matter how long it 
takes a student to learn something? If a student- does take a 
little longer to learn, but still learns, why not give that, 
student full credit for learning?" 

ERJC 



In. mastery learning, a much higher premium is put on 

degree of learning than on rate of learning * So most students 

can master each lesson as it comes. This is possible because 

each student is given the time he or she personally needs in 
order to learn. 

Learning isn't threatening or painful in mastery learning 
Students know they can learn and know how to learn. Several 
studies have shown evidence of increased student achievement in 
mastery situations. When students realize that they can learn, 
through reteaching materials and retesting if need be, they 
usually decide they want to learn, and put more effort into 
learning. 

It's important for students to feel they have some con- 
trol over their learning. Many adult low level readers, espe- 
cially, have bad feelings about learning due to past coercion 
and punishment in school. Mastery learning allows the student 
to learn at his or her own pace. And instead of the teacher t 
always being the one with the questions, students also ask ques 
tions and take responsibility for their own progress. 

Finally, mastery learning benefits the whole class be- 
cause students often learn together, not as a collection of 
competing individuals. In mastery learning, students compete 
against themselves for mastery, not against each other for 
grades. Students cooperate and help each other because they no 
longer fear that they will lose special advantages over their 
classmates in the competition for grades. 

« 

The teacher's role 

In the traditional classroom, the teacher is primarily a 
lecturer. In the mastery classroom, the teacher's role is more 
that of a consultant. 
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Under mastery learning, the teacher: 

1. Introduces students to what is to be learned, ex- 
plains a few things about mastery learning, and explains how 
to use the materials. 

2. Answers students' questions and helps them when they 
need help. Usually, the teacher helps students individually. 
But sometimes, several students will be at about the same place 
in the materials and will be having the same problems. Then the 
teacher will want to group these students and teach them all to- 
gether Cfor that one particular problem). If there is something 
which most students have had trouble with, the teacher may want 
to give an explanation to the whole class at once. 

3. Always has the option of supplementing the mastery 
materials with various other activities. Some of these might 
be doing extra worksheets, reading extra review material, class 
discussions, role-playing, class trips, and so on. 

4. Administers the materials at a rate suitable to each 
student. 

5. Helps students keep track of their progress. 

6. Praises, encourages, motivates. 

7. Lectures and demonstrates when most students have 
mastered a particular section. The teacher doesn't give tests 
on the content of these lectures and demonstrations. Their 
only purpose is to motivate students and stimulate learning. 

One big advantage of mastery learning is that teachers 
have the opportunity to give students individualized attention. 
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Some teachers have very strong preferences for teaching 
their class as a whole. In mastery learning, that can work 
like this: 

« 

a. The teacher has the whole class study a unit together, 

b. Then each student takes a test on that unit, 

c. Students who don't pass the unit study reteaching 
materials, then take a second mastery test. 

d. Students who do pass the unit either tutor the others, 
do free reading, do reading or other activities related to the 
topic of" study, do work for other courses, or work on a personal 
project. 

e- When all or nearly all students have passed the mastery 
test, the class goes on to the next unit. 

In general, we feel that it's better to allow each student 
to move ahead at his or her own pace. Most students, once they've 
passed a mastery test, don't like being held back. Still, the 
"whole-class" system is a viable option. 

Mastery tests 

Mastery tests are " criterion- referenced" tests. They re- 
fer or compare student performance to a standard or criterion 
of proficiency in a given skill or knowledge area. If a student's 
score surpasses the cut-off "mastery" level on that criterion, 
then the student is said to have "mastered" that skill or knowl- 
edge area. 

This is significantly different from achievement or " norm - 
referenced" testing. Here the student 1 " performance is not com- 
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pared to an absolute standard or criterion of proficiency. 
Instead, the student f s performance is referred or compared to 
the performances of fellow students or to the norma l curve 
distribution of performances of students of the same age or 
grade Xevel across the U. S. 

i 

In mastery learning, students take a mastery test after 
studying each unit. These tests are for the purpose jof guid - 
ance , not grading. They tell the student if he or shp knows 

the material, or if extra study is needed. Also, wrong answers 
on specific test items indicate specific deficiencies! 

Mastery tests should test for different types of knowl- 
edge. An overemphasis on one type, say "definitions" j *:ould 
lead students to not develop other types of thinking skills. ' 
It f s a good idea, even before developing the first test, to list 
the types of skills to be included in the tests. These should 
be skills which you know the particular users of the material^ 
have the capacity to learn. 

Mastery tests can give a school, and employers outside a 
school, a good idea of what a student actually has learned. 
When a student demonstrates mastery of a course, mastery becomes 
a legitimate form of certification. Grades based on class rank, 
on the other hand, are difficult to interpret. 

Testing and grades 

Limiting the length of time for learning a given unit of 
material keeps slow students from learning thoroughly. In a 
sense, it discriminates against the slow student. The discrimi- 
nation is even greater if slow learners are consistently given 
ldw grades. These low grades label slow learners as inferior. 
.And this labeling generally has a negative effect on the stu- 
dents 1 attitudes toward themselves. 

26 



There are other serious problems with the traditional 
grading system. For qne thing, grades have no meaning across 
classes or across schools. Grades just rank students within a 
class. So an A student in one school could, with the same per- 
formance, conceivably be an F student in another school. 

Another problem is that student's grades depend on how 
other students ill the class do. So if a student's performance 
improves, but pther students 1 performances also improve, the 
student's grade stays the same. 

In mastery learning, testing and grades are handled dif- 
ferently. Mastery tests compare the student's current perfor- 
mance with his or her own prior performance. The tests don't 
give the student, negative messages about how he of she is doing 
compared to other students. 

In mastery learning, the number of rewards isn't limited. 
As long as, a student learns to the level expected, he or :he is 
given credit for succeeding. One can compare fast and slow 
students to motocycles and bicycles. One may be faster than the 
other, but they can both get there. 

Performance standards 

There aren't any hard and fast rules for judging mastery. 
I* -6ally depends on the curriculum writers', teachers', and 
administrators' perceptions of what would constitute mastery in 
each instance. Difficulty level will vary a bit from unit to 
unit end test to test. So for on* unit, an 804 level of correct 
or appropriate responses might be sufficient, for, another - 90%, 
for another - 70*. 
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Reteaching materials 

When students don f t reach the mastery level on a test, 
they are given lf reteaching" materials to bring them up to 
mastery level. Reteaching materials teach the same material 
as the original instruction, but from a different perspective. 

The simplest reteaching is the help a student can get 
from a tutor, small group discussion, or the teacher. For your 
materials, there are many options. Among them: 



1. an alternative textbook (deliberately designed to 
correlate and complement the Main Text in " enhanced" 
mastery learning) 

2. f drill sheets 

3. a consumable workbook 

4. audio-visual materials 

5. flashcards 

6. games and puzzles 



7. activity cards 



8. a combination of the above 

In general, reteaching materials don f t seem to work well 
when there are too many items to keep track of. Above all, the 
reteaching materials should be manageable. Teacher-consultants 
can be useful here in helping you judge what will easily work in 
actual classrooms. 
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Other advantages of mastery learning 

Problems mastery learning can solve: 

1. Frequently absent students cannot keep up with the class. 

With mastery learning, a student resumes learning where 
he or she last left off. Students don f t have to feel embar- 
rassed or lost when they come back from an absence. And be- 
cause mastery learning is less painful, normally poor attenders 
may not be averse to coming to school. 

2. Some students have had numerous failure experiences. They 
may have just given up and no longer try. 

With mastery learning, students start passing checkups 
from tht first day on. It doesn't take them long to real&e 
that their efforts bring success. In our field tests, we ob- 
served many "poor" students trying hard and succeeding • Stu- 
dents who'd never done well before started getting perfect 
scores Cgranted, perfection did not always come until their 
third mastery test for a unit, but perfection is perfection). 

A friend who tried mastery learning, a teacher, told us 
this story. A black girl in his ninth grade class had never 
gotten higher than a C in a class. She'd tried to learn, but 
she was always behind. She tried especially hard in his mas- 
tery class. The last day of class, she came up to him and 
asked if she'd finally earned a B. He told her no. She just 
sighed and said "oh". Then he said, M Ybu got an A." The 
girl's face quivered and then she broke down crying. The A 
gave her an image of herself she'd never even dreamed of 
before. 

3. Many students daydream or misbehave whije the teacher lec- 
tures. These students are used to being active people, and 
they don't like the passive role. 
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, In mastery learning, students are always busy. They're 
always responding. Not only that, there's less of a feeling 
of always taking orders from the teacher. If the student gets 
tired of studying the mastery materials, he or she can take a 
short break. Other times, when the stud feels like speeding 
along with the materials, that's possible too. (And don't be- 
lieve that won't happen. This comment, from a student in our 
field test who was busy at work, might give you an idea: "Hey, 
don't let that bell ring. I'm really hot now!") 

4. With classes of adult low level readers, there's usually a 
wide variation in reading ability among students. 

This doesn't cause a problem with mastery learning. The materials 
can accommodate a wide range of abilities. Also, with individuali 
ration, students can learn at whatever speed is most comfortable. 

5. Each of these learners has unique learning problems. In tra- 
ditional instruction, if a teacher takes much time helping 
one student, the class stops and the others have to wait. 

In the mastery system, learning isn't so dependent on the 
teacher. If the teacher takes time to work with one student, the 
other students can keep learning on their own with the materials. 

6. Many of these students have bad learning habits. They don't 
make a second effort if they fail to understand, they read 
without trying^ to comprehend, they don't read anything unless 
the teacher tells them to. 

Mastery learning teaches students how to figure things out. 
And it teaches them that they don't always need the teacher in 
order to learn. 

7. Few publishers put out good materials for this audience. 
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With mastery learn. g t you 
(by writing reteaching materials 
fit the needs of your students. 



can adjust existing textbooks 
and mastery tests) so that they 



A Day in a Typical Mastery Learning Class on Bank Checking 

It would take hundreds of pages to describe, in detail, 
how actual enhanced mastery learning courses have worked. But 
we would like to at least give you a taste of what it's like to 
teach such a course. 



For this example we assume a very small class (to keep 
this from becoming too confusing for the reader) ot four stu- 
dents: Mike, Betty, Sandra, and Carlos. Read the description 
on the left, then look at the right-hand column which gives 
you-*n— idea-of Jiow . _the_teacher moves around through the ma- 
terials. 



Description 



Teacher Preparation Faction 



The teacher, Ms. Wallen, ii 
now ready to. have her class be- 
gin lesson 3. She begins her 
class with a lecture. This lec- 
ture describes briefly the topics 
to be covered in Lesson 3. 



She studies the teacher's 
Guide and the contents of 
Lesson 3. 



The first topic in Lesson 3 
is Deposit Slips. 

Sandra is the student who's 
been moving most quickly through 
the materials. But lately Carlos 
has been learning quickly too. 
Ms. Wallen decides to start these 
two off in the*green book.* 

Betty seems to have been 
doing well in the blue* material 
lately, so Betty will begin in 
the blue. 



Looks at the descriptions 
of the enrichment, main and 
reteaching texts in the 
Teacher's Guide. 

She looks at her map of 
student progress. Using this 
map, Ms. Wallen determines 
that Sandra and Carlos should 
work in the Main Text, page 17, 
Betty in the enhancement text, 
pages 20-21, and Mike in the 
reteaching text, pages 7-9. 
Note: the information about 
each student appears on that 
student's Progress Sheets, 



For simplicity, we'll say that: "the blue book" - "the Enrich- 
ment Text" 
"the green book" - "the Main Text" 
' "the yellow book' - "the Reteaching Text. 



31 



15 



Mike has had some problems 
with both the blue and the green 
materials, so Ms. Wallen decides 
to start him off in the yellow. 



in his/her folder. So stu- 
dents can quickly see where 
they should be. 



Sandra finishes first. She 
asks Ms. Wallen for a' check-up. 
Ms. Wallen checks her exercise 
on deposit slips in the green book, 
*sks her an informal question 
about deposit slips, then hands 
her a check-up (mastery test). 



Ms. Wallen uses her answer 
key to check Sandra's work, 
and turns to her Deposit 
Slip check-ups. Ms. Wallen 
decides to give Sandra 
check-up #001. ♦* 



Now Carlos has finished. 
Carlos still doesn't quite 
understand the wordr "endorse. 9 ' 
He asks Ms. Wallen about it 
and she explains. Then she 
gives Carlos a Deposit Slip 
check-up. 



She gives Carlos Deposit 
Slip check-up #002. (She 
doesn't want him to have the 
same test as Sandra.) 
Note: for each lesson, the 
teacher has four or more 
alternate but equivalent 
Mastery tests, or different 
versions of the same test. 



Betty says, "Hey, Ms. 
Wallen, I've finished page 21." 
Ms. Wallen says, "Okay, hang 



on a second. I f ra 
helping Carlos.' 9 
comes over. 



almost done 
Then she 



Ms. Wallen looks up 
answers in the blue answer 
key, then gives Betty De- 
posit Slip check-up #003. 



Ms. Wallen Checks Betty's 
exercises in the blue. Betty 
missed the first question. Ms. 
Wallen asks her about it. It 
turns out to be just a minor 
confusion. Now Betty under- 
stands. So Ms. Wallen de- 
cides to give her a check-up. 



Sandra asks Ms. Wallen 
to score her check-up ("Deposit 
Slips check-up #001). 



Ms. Wallen turns to the 
Deposit Slips #001 Score 
Key. 



Ms. Wallen finds that Sandra 
has made no mistakes. She asks 
Sandra a few off-the-cuff ques- 
tions and finds out that Sandra 
really seems to understand de- 
posit slips quite well. She 
winks at Sandra and says, "Hey, 
nice goin 9 , kid." Then she gives 
her a new assignment. 



Looking at the bottom of 
the Deposit Slip #001 score 
key, Ms. Wallen decides to 
assign green pages 18, 37-42. 
She circles those pages under 
the green column in Sandra's 
progress sheet. 
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Mike wants Ms. Wallen to check 
his exercises. She does, and she 
finds that Mike has made several 
mistakes. She discusses the cor- 
rect answers with him. Then she 
gives him a check-up. 



Ms. Wallen looks up the 
answers to the exercises in 
Checking Illustrated (the 
yellow material j . Then Ms. 
Wallen gives him Deposit 
Slip check-up #004. 



Ms. Wallen checks Carlos' 
check-up. Carlos got both the 
cash amount and total amount 
wrong. They chat a moment about 
the problem, .then Ms. Wallen says, 
"I think the blue materials would 
be help here, Carlos." Then she 
assigns him some "enriched" re- 
teaching 4 n the blue materials. 



Ms. Wallen turns to De- 
posit Slip score key #002. 
Using the map at the bottom, 
Ms. Wallen assigns Carlos 
pages 20-71 in the blue ma- 
terials. She circles these 
pages in his progress sheet. 



Betty also makes several mis- 
takes on her check-up. Ms. Wallen 
assigns her reteaching in the 
yellow materials. 



Using Deposit Slip score 
key #003, Ms. Wallen assigns 
yellow pages 7-9b. 



Sandra needs a new check-up. 



Ms. Wallen gives her the 
Check-writing check-up #001 



Ms. Wallen checks Carlos' 
exetcise answers in the blue 
materials. He got all answers 
correct. She then gives him a 
check-up. 



Ms. Wallen turns to the 
exercise answers for Under - 
standing Checking (the blue 
material) . 

Ms. Wallen gives Carlos 
Deposit Slip check-up #001. 



Ms. Wallen checks Mike's 
check-up. He's still making 
several mistakes. She assigns 
him some reading on deposit 
slips in the blue material. 



Ms. Wallen uses Deposit 
Slips score key #004. She 
assigns Mike blue pages 20- 
21. 



Let' « skip ahead a little. 
Carlos has now mastered the unit 
on deposit slips, whereas Mike 
still isn't clear on a few things. 
Ms. Wallen asks Carlos if he'll 
help Mike understand deposit slips 
Carlos agrees. 



Mike is in blue pages 
20-21. 



Both Sandra and Betty have 
now mastered deposit slips. 
(Actually, Sandra is already well 
on her way to mastering check- 
writing.) Ms. Wallen suggests 



Ms. Wallen is using 
activity #1 under remedial 
activities, and #2 under 
enrichment activities in the 
Teacher's guide. 



ERIC 



33 



17 



the two" of them role play opening 
a checking account. She also 
suggests they fill out deposit slips, 
depositing ten checks 



Mike takes a second check-up on 
deposit slips, and passes this tine.. 



Carlos asks Ms. Wall en if he 
could try the yellow materials for 
the check writing section. He likes 
the cartoons. Betty wants to try 
the green fo** a change. She says, 
"You learn faster in the green." 
Sandra want* to try the blue. She 
finds it rore interesting than the 
green. I*r.~1fallen agrees to all of 
these requests. 



Ms. Wallen gives him 
Deposit Slip check-up #002. 



Ms. Wallen uses the stu- 
dents' progress sheet to 
show them what pages to study 
in the books they've selected. 



This should begin to give you a feeling for how the system^ 
works (at least under this particular management strategy) . 

One important difference from usual classroom methods is that 
teachers do little lecturing. The teacher is a manager of the 
instruction. The materials shoulder the primary burden for the 
instructing. The teacher administers, helps special individual 
problems, and fills in the gaps where the materials fall short. 

Most of the teacher Y s teaching in mastery learning comes at 
three points: 1) when students ask questions or look confused, 
2) when the teacher corrects answers on the materials and check- 
ups and the particular student needs explanations for items 
he or she missed, and 3) when the teacher feels the time is right 
for additional teacher-directed activities. 

It is our experience that the first few days with the kit 
are by far the hardest for the teacher and the class. It takes 
a little while to get the class into the swing of things. But 
once everyone gets used to the system, you'll be pleasantly sur- 
prised at how smoothly the learning goes -- and by how well- 
motivated your students become* 
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PART TWO: DEVELOPING A MASTERY LEARNING SYSTEM: KEY DECISIONS 
AND ACTIONS 

STAGE ONE: PRE-PLANNING DECISIONS 

Assumption: you are ready to consider the wisdom of 
building a mastery learning system. 

I. Do you really need a mastery learning system? 

A. Do the materials in your instructional system really 
need to be improved, or can your resources be better 
spent on improving the teaching, organization, or 
student -support in your system? 

- • 

B. If you do need better materials, should they do the 
following for your most-in-need students? 

1. Offer them the opportunity of a second, or even a 
third, chance to restudy, relearn, and be retested 
on a lesson until they achieve mastery? 

2. Test them in order to decide whether they, have 
mastered the current lesson and are ready to go 
on to the next lesson? ^ 

3. Test them on whether they have mastered the 
material, and not on how they compare to fellow 
students or to a» national sample of students of 
the same age or grade level? 

Enable them to work and learn by themselves with 
a minimum of help from their teachers? 

Enable them to learn and move through the materials 
at their own pace regardless of how fast the rest 
of the class is moving? 

35 
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If you answered "yes" to all of the above questions, then 
you may need to build a mastery learning system for your 
most-in-need students* (Even if you answered "no" to #4 
or #5, you may still need to build a modified version of 
such a system. ) 

II . Can you afford to build a mastery learning system? 

A* Will your teachers accept mastery learning or too 
strongly resist it? 

B. Will key administrators endorse building the system 
and, later, encourage teachers to accept and use it? 

- C. To what extent will your current materials need to be 
revised, and/or new materials and tests need to be 
developed? At what cost?* 

D. How thoroughly will you need to field-test the relia- 
bility and validity of the tests? At what cost?* 

E. Can gaps in your staff's expwrtise be filled by avail- . 
able and qualified outside experts? At what cost?* 

F. How much will C.-E. Cost? Will the expected benefit 
from a new mastery learning system be greater than 
this cost? 

G. Will the needed funds and staff time in fact be avail- 
able in the amounts required during the development 
period? 

If you answered "yes" to both F and G., then your organi- 
zation should build a mastery learning system unless another 
strategy will produce even greater benefits at the same or 
lower cost level. 

x " 

All you can do at this stage is make "ball park" esti- 
mates of these costs. Read this entire Guide before attempting 
to estimate development costs. 
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STAGE TWO: PLANNING AND PREPARATION 

Assumption: you are virtually convinced that you should 
and can build a mastery learning system. 

A. Recruit a director for the development project who 
Will champion, manage, and commit to it. 

B. Determine whether your most-in-need students need 

4 

their learning materials to be (listed roughly in 
order of increasingly ambitious improvements to be 
undertaken) : 

1. Supplemented with mastery tests for each unit or 
lesson. 

2. Supplemented with more exercises and practices. 

3. Supplemented with additional instruction to fill 
in gaps in the current instruction. 

4. Revised by being up-dated and made more relevant- 
and interesting. 

5. Revised by breaking down the instruction into 
smaller steps or units that teach underlying 
skills and subskills. 

6. Revised by being rewritten using simpler vocabu- 
lary/constructions . 

7. Supplemented with a full-blown alternate text that 
could teach the whole subject by itself but teaches 
or reteaches the subject in a very different way. 

8. Replaced by completely new materials using an 
« entirely different instructional approach. 

9. Combinations of #1-8. 

(NOTE: throughout the rest of this Guide, we will 

assume that you need a combination of improve- 
ments #1 - #7.) 

37 
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C. Based on this needs assessment, describe the size, 
scope, and nature of the products to be developed 
in the project. 

D v Based on this description, determine what steps and 
processes must be undertaken to develop those pro- 
ducts; determine staffing requirements. 

E. Specify the degrees of reliability and validity which 
^the tests must be proven to have; if these require a 

field-test, estimate' its cost. 

F. Determine the actual availability and cost of outside 
* consultants and services to backstop or fill gaps in 

expertise (e.g., expert teSt developer, evaluator, 
outside printer, artists, etc.). 

G. Refine the Step One estimate of funds, staff, outside 
consultants, and time needed to carry out the develop- 
ment project. Re-evaluate the benefit-cost ratio. If 
the project no longer appears to be cost-beneficial, 
abandon this effort. Otherwise: 

1. Recruit and organize the development team (instruc- 
tional developers, writers, test developers, (evalu 
ator if field-test needed), administrators 1 , and 
support staff) . j 

2. Invite and obtain commitments to the project from 
those in the environment whose acceptance will in- 
fluence whether the project will be a success 
within your organization. * 

3. Establish guidelines about the process for making 
management decisions, materials-development de- 
cisions, and test-development decisions. 

4. Get commitments -to- cooperate from those teachers 
who you will want to try out early drafts, of the 
materials and/or tests. 

38 



Develop a calendar of development milestones or 
deadlines by which time each phase of the project 
should be completed. 

Obtain formal commitments of funds and staff 
needed to launch. 



38 



23 



APPENDIX 



, Initial contacts with teachers and students 

When you're sounding out interested teachers who might be ' 
willing to participate in the pilot-test, you should if 
possible: 

- Observe teachers and students at potential sites 

* Take note of reading abilities, study skills, materials 
preferences of students 

— Take note of how teachers manage their classes as well as 
what materials they prefer and how they like to present a 
subject. 

I * a 

Such observations could tell you if certain sites would not be 
appropriate for the kind of materials you are considering 
developing. For example, in ou^r project if we had observed the 
two English for Speakers of Other Languages programs in Syracuse 
before we pilot -tested materials in them, we would have realized 
that our materials were inappropriate J;or them. The poor match 
was painful both for those teachers and students and for us 
during the pilot- test. 



Qualifications of key members of the development team 
Director : 

- Able to act as team builder with staff that has di- 
verse abilities. Helps each team member to see the 
value of the others, to learn to depend on the others, 
and to enhance the work of the others, without stray- 
ing beyond their sphere of greatest usefulness. 

- Able -to win cooperation of administrators and teachers 
at pilot (and field) test sites, i.e., clearly ex- 

erJc . 40; ; , 
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plains nature and scope of testing program and its 
usefulness to the students and teachers involved. 

' ' i 

- Able to keep sponsors .informed about project's 
progress. 

- Able to map out tasks to' be accomplished, by whom, . 
and by when, in order to reach projects objectives. 

- Able to project costs of all activities. 

- Able to monitor each activity of the project to make 
sure that agreed-upon quality and quantity are de- 
livered by agreed-upon deadlines. 

! 

Materials Developer : 

- Is aware of contemporary theories of instructional 
design, knows which would be appropriate for the 
target audience and can employ appropriate design 
models in writing the materials. 

- Able to write at the reading level of the audience. 

- Knows the subject area. 

- Has experience in teaching the target audience or in 
observing such teaching. 

- Writes concisely, simply, concretely, and interest- 
ingly. 

Test Developer ;* 

- Has background in instructional design and test and 
measurement for task analysis and the writing of 
learning objectives. 

This is th* most critical position in regard to schedule 
and quantity of final product. Make absolutely sure you enlist 
someone who knows what he/she is doing, is highly productive, and 
can meet deadlines. A slow test developer will slow down or wreck 
your whole project. 
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Has practical experience in developing test items 
of a variety of types and can present a portfolio 
of past work. 

Understands the difference between norm- referenced 
and criterion u referenced tests and can demonstrate 
competence regarding key issues such as deciding 
test length, test frequency, test format, teacher 
administration and scoring of tests, etc. 

Has statistical background and experience in item 
analysis, item discrimination assessment, reliability 
assessment, and validity assessment for both cri- 
terion and norm referenced test development. 

Can collaborate with others and share power and 
responsibility ?.a a teamwork environment. 

Has experience in designing and conducting 7 pilot 
tests of instructional materials (observing students, 
interviewing teachers, etc.). , 



All the above qualities of the materials developer 
(except that the editor need not be quite as knowl- 
edgeable as the author about the subject area). 

Able to catch inconsistencies in content, word usage 
and matters relating to particular style of presen- 
tation adopted by publisher, e.g., rules on when to 
italicize, use bold face, underline, capitalize, use 
parentheses, use hyphens, etc. 

Able to use readability formulas and re -write ma- 
terials without sacrificing the human interest and 
accuracy while making it easier to read. 
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- Able to visualize final page -- where illustrations 
will go, how they'll be integrated with text, how 
wide the column width should be for text, how large 
illustrations will be, etc. 

- Able to communicate this vision to artist/ layout- 
designer. 

- Able to suggest content of illustrations in way that 
inspires artist to find precisely the illustration 
that communicates the idea to be taught. 

- Able to direct the artist toward desired design without 
stifling the artist's creativity. 

Artist : 

- Has experience directly or indirectly with target 
students in some learning situation, and is sensitive 
to their tastes, values, or ethnic composition so that 
her art expresses knowledge and appreciation of their 
culture. 

- Knows the kinds of illustrations target students (in 
general) appreciate across a range, e.g. stylistic 
cartoons, photos, simple diagrams, but no straight 
line drawings, complicated diagrams (editor may have 
to communicate this to the artist). 

-, Respectfully portrays representatives of the target 
student population including differences in age, sex, 
ethnicity. 

- Able to grasp the idea which the editor wants illus- 
trated and can elucidate it. 

- Willing to put learner's needs for clarity and 
simplicity above artist's desire for cleverness. 
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Consultants and the State-Of-The-Art 

Consultants provide invaluable help in the development 
of mastery learning kits. You may need consultants who are 
expert in-- 

1. The subject you're presenting 

2. The development of mastery tests 

3. The development of teaching materials 

By reading the latest arnals and books in the area of 
instructional design, you'll find names of those who are promi- 
nent in designing instructional materials and criterion- . 
referenced or domain- referenced tests. Contact the ones who 
seem to be doing the kind of work you want to have done. Seek 
out the very best ones first. Their pried tag may be too high. 
If so, you can ask them if they know of others who could do 
the work. Try to find someone close at hand, for instance at 
a near-by university if there's one in your vicinity. 

Do not contact anyone until you have familiarized your- 
self with the literature and the "state-of-the-art. ff Consul- 
tants work for you . They cannot lead, direct, or make decisions 
for you. They are most comfortable and effective when they are 
handed a well-defined statement of the scope\f,work they are 
expected to perform. You cannot formulate such a statement if 
you are ignorant of the state-of-the-art in the less technical 
aspects of mastery learning and criterion-referenced testing. 

Start with this little Guide and read items referenced 
herein. Th i locate three or four articles on mastery learning, 
and three or four on criterion/domain- referenced testing, from 
journals or magazines published within the past three years. 
This much research on your own should suffice and will pay back 
big dividends in the future. 
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Examples of Planning Aids We Used in Our Development Project 



* PERT MAP 



* GANTT Schedule of Activities 
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8CHBMMI O AarxvAXuB 
Initiation Date: October 1, 1980 



OCT 



NOV 



EEC 



JAN 




JUL 



AUQ SEPT 



M - 

A-10 
C-4 



Ttm instructional (reteaohlng plan) system in the Kit is 
appraised and prototypes are revised in the litfit of the 
pilot test. That system and the physical structure of the 
Kit are improved yielding two "best" prototypes to be 
ocapared in the field test. 



The Mid-Tera Progress Report is prepared and submitted to 
the CEP project officer. 



A dissemination plan is developed aid steps are taken to 
secure nailing addresses of those audiences to whom 
announcements will be sent. . 



The docurentation of the test-development and kit-develop- 
ment processes is reviewed and a methodology is desigred 
for evaluating the processes in order to create a 
replication manual which will enable others to easily and 
Inexpensively replicate or emulate those processes. 



POUR- 

TEE* 
7IF- 
THs* 
M- 
A-10 
C-5 



The two "best"* prototypes of the "MP-OUC are finalized. 
Arrangements with field participants are finalized. 
Participating teachers and tutors attend a workshop to be 
trained In procedures for recording and reporting 
evaluation data. Students are taught the "Iff materials 
and are tested as designed. Teachers and tutors and 
students are interviewed/observed. Results are assessed 
and the best features of the two prototypes are synthe- 
sized. Pinal development of the "W-QUC. 



SIX- 
TEENl 



A report is prepared describing the criterion-referenced 
tests* technical characteristics. 



The process docunentatlcn is evaluated and the Model Is 
developed. It guides the creation of the Replication 
Manual which la cospleted and 100 copies are printed. 



EE 



Eftsaaadnatlcn amouncemsnts are mailed, responses are 
tabulated and acted upon. 
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The Filial Report is prepared and submitted to CEP. 
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NINE 



1-8 



TON 
SE- 
VEN 



SCHEDME OP AuiXVXTJES 
Initiation Date: October l t 1980 



A search Is conducted far potential participants In the 
field test beyond those agencies contacted In the course 
of preparing this proposal; the search Is guided by the 
requirements of the sanpUng plan. 



Public comnunlcaticns and education announcements are 
developed In an effort to Involve the conmunity and to 
raise public awareness of the Issues addressed by the 
project. 



Items are 4 generated for possible Inclusion In the 
criterion-referenced mastery tests. Scoring methods are' 
designed for the Items, • 



Hie Kit's system of reteachlng plans coordinated with 
possible outcomes of administrations of the criterion- 
referenced mastery tests Is further developed.. 



The generated test items are subjected to examination 
for content soundness and fidelity (to establish validity) 
and for technical soundness. 



OCT 




With the cooperation of appropriate agencies In central 
New York, five students are Identified and Invited to 
participate In the pilot test; selection Is guided by the 
sanpUng plan. 



B-5 
A-9 
A-10 



The B-4 configurations of the best features of other kits 
are modified In the light of the Project's other test and 
kit development activities; three or four "realistic" 
configurations are built Into prototypes to be evaluated 
In the pilot test. Directions for teachers and decision 
rules guiding master/nonrmaster classifications are 
developed and Incorporated Into the prototypes. 



TWE- 
LVE 



The pilot test evaluation Is designed, carried out, and 
Its outcomes are assessed and used to eliminate defects 
and guide the development of two "best" prototypes to be 
compared In the fieJ* test. ' _ 
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Arrangements are made with cooperating schools with 
students comparable to the target population to carry out 
the assessment of the reliability of the mastery test item i 
Students are tested In accord with a reliability assess- 
dulgn developed by the Evaluates* and Dr. Gardner. 





SBBP 
OR 
Task 
No. 


SCHEDULE OP ACTIVITIES 
Initiation Date: October 1. 1980 


OCT 


NOV 


EEC 


JAN 


FEB 

• 


MAR 


APR 


MAT 


JUNE 


JUL 


AUG 


1 

SEPT 




ONES' 


Project Director convenes participants to review and ] 
revise the Project plan. 


L 










- 


















Director meets with CEP project officer to clarify any 
anbigultles, discuss specifics, and develop an under- 
standing and routine process regarding Federal oversight. 


▲ 
























mo 


Develop preliminary blueprint of Instructional objectives 
embedded In Ill's "Money Management" curriculum. 


A 

m 


A 


















• 




B-l 


Description of the Intended "Money Management "-Guided 

JJBalTlinK IMu SulbctOie I OP pUDJJLC (XxiliRXlXCaUUllo IS 

developed. 


A 




















- 




3-2 
J-3 


Developers and publishers of conparable programed 
Instruction, mastery learning, and related systems are 
Identified, contacted, and asked to let III borrow or 
buy copies of their best and most recent products. These 
are examined and evaluated and their best features are 
Identified (as are their cost-related aspects). 


A 




i 




















■ 


The "Money Management" instructional objectives are, 
transformed Into "descriptive fonrf 1 . Skills students 
should have before entering the "Money Management" 
currlculun are preliminarily identified. 


■ 


1 






















•OUR 


The "Money Management" (VH) objectives are prioritized. 




















JL 


3-1 
3-2 


Key Issues regarding the Kit's physical structure are 
Identified and plans are made for finding solutions. 


























The foundation for the Kit's system for re teaching non- 
master students the objectives they have not mastered. 




























The test blueprint Is validated. 








A 


















B-H 


From the best features of the exenplary kits and systems 
obtained from other publishers, 3*5 configurations of 
their best features are designed. 




















• 






SIX 

even 


Domain. specifications for all objectives are developed, 

and a Y*Ar\crt* nf nivihahlv acgp*o table test lengths is 

estimated. 


























C-3 


Physical specifics are specified for the B-4 configuration 

<3 
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STAGE THREE: PREPARE FRAMEWORKS AND MODELS 

ASSUMPTIONS: Organization and funding are in place 

Long-Run. Goal is to carry out improvements 
#l-#7\in Part B, Stage Two. * 

A. Project Administration \ 

\ 

\ 

1. Form and orient a Project Review Corimittee composed of 
those administrators, and t6p staff who should be peri- 
odically briefed on the Project's progress.. Arrange 
schedule of review meetings. 

\ " • 

2. Form and orient a Mastery Learning Panel composed of 

those staff and professionals who are responsible for 
the course targeted for development or improvement. 
Define relationship to Project Team and. their mutual 
scopes of responsibility. If the course will be brand 
new, this Panel is responsible for defining its goals 
and scope. 

3. Refine the development plan to accord with the scheduling 
I realities, personalities, and limited resources the Pro- 
ject will have to deal with. v 

4. Identify short and medium-range development targets or* 
deadlines and get Project Team members to commit to 
attaining them .on time. 



B. Field Work . 

1. Finalize the identifications of the (kinds of) students 
who are intended to be the ultimate beneficiaries of the 
Project's developed or improved materials. 

2. Select two (or three) sets of classrooms that are repre - 
sentative of the targeted student population ("target 
audience"). One set will participate in the Diagnostic 
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Preview, another in the Pilot Test (and the third in 
the Field Test if there is to be one). Contact the 
teachers of these classes and obtain their commitments 
to participate. 

3. Carry out t ; Diagnostic Preview: all Project Team 
members visit the first set of classrooms for a few 
days to observe how those classes normally use exist- 
ing materials in or related to the course to be im- 
proved or developed. Team members discuss advantages 
and shortcomings of existing materials with students 
and teachers. Findings become part of the basis for 
developments of teaching and testing materials. 

Test Development and materials Development (Joint Effort 
in this Stage) 

1. Test and materials developers work together in analyzing 
the existing materials to be improved (or the goals for 
the course to be developed). With the Panel, they 
identify what it is that the course and materials are 
intended to teach. 

2. They develop models of actual Learning Objectives, the ^ 
Blueprint of earning Objectives, Domain Specifications, 
tc^sts, and teaching materials to be developed in the 
Project. They work fairly closely with the Mastery 
Learning Panel. 

a. Fc v existing materials, the developers decide where 
alx possible appropriate breaks are for inserting 
mastery tests. 

b. For a new course and/or new materials, decide how 
the teaching of the instructional objectives should 
be sequenced, and how this sequence should be chunked 
(how many objectives should be taughc in each unit or 
lesson) . 
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c. Decide on the optimal length for the average test, 
how many tests there will be in all, and how many 
items are needed to* test, or measure mastery, of 
each objective. 

d. Develop a model domain specification for one or 
more objectives identifying performance criteria, 
conditions, and types of items needed. 

Obtain and examine exemplary mastery learning kits from 
around the U.S., and locate and read books on instruc- 
tional design. Identify the best features of the kits, 
and select the best ideas from the instructional design 
resources. Develop preliminary ideas of methods; for 
reteaching each lesson and consider alternative format 
styles. 
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STAGES THREE AND FOUR TEST DEVELOPMENT 

» 

INTRODUCTION 

The goal of the test developer is to generate valid cri- 
terion-referenced tests (CRT f s) which teachers can use to 
separate those I students who have mastered the instruction's 
objectives (and should advance to the next lesson) from those 
students who have not mastered the material and need further 
work. 

A CRT f s validity is its ability to measure the student f s 
mastery of an instructional objective comprehensively with 
fidelity . The items in a valid test should elicit behaviors 
from the student which enable an observer to judge whether the 
student has or has not learned the instructional objective's 
competencies 9 knowledges 9 and skills. The items in a valid 
test should measure the instructional objective with fidelity 
(the test should contain only items which are relevant) and 
those items should compr ehens ive ly measure the student's pro- 
ficiency in all of the competencies f knowledges, and skills 
subsumed under the objective. 

Comprehensiveness can rarely be perfectly attained. There 
is no mechanical rule which will automatically break down the 
typically vague instructional objective into/iall of its component 
competencies 9 knowledges 9 and skills. Yet th^ test developer 
should try to ensure that a test f s items genuinely relate to all of 
the proficiencies which the test is intended to measure. One way 
to come as close as possible to perfect comprehensiveness is to 
involve several individuals whose blind spots will cancel out. 
The CRT test development procedure described here recommends 
using several judges who complement each other in trying to en- 
compass all competencies to be tested under each instructional 
objective. 

A test lacks fidelity if its items measure proficiency in 
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one or more subskills that are not necessary or essential to 
the mastery of the objective. Individuals will disagree about 
what is necessary and essential, about what one must know or be 
able to do to be called a ff master ,f of a particular objective. 
Again, the procedure described here recommends using several 
judges whose biases will cancel out. 

1 # Develop an outline of the subjects or course's objectives* 
(Although it is desirable to formulate these as specifically 
and behaviorally as possible, the goal here is a comprehen - 
sive and complete listing of the course's objectives.) 

Form the Mastery Learning Panel composed of all those who 
either participated in the development of the materials or 
are responsible for the nature of the courses. 

2. The materials and test developers in the,. project indepen - 
dently examine the current materials and identify the com- 
petency, knowledge, and skill objectives they teach. 

3. The two developers compare and combine their lists, and 
they identify objectives that seem to be candidates for 
deletion, combination, and refinement. 

4. The developers submit the combined list to the Mastery 
Teaming Panel; new objectives for the course are added, 
unwanted objectives are deleted. 

All work together to enhance the list's comprehensiveness 
(everything to be taught by the course is represented or 
"captured" by at least one objective) and fidelity (no 
objectives test things not taught somewhere in the course). 



For more on objectives, see tbc Appendix at the end of 
this section. 
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5. The test developer then translates each objective into state- 
ments that are as behavioral as possible (what you would see, 
hear, etc. if you observed a true master of the objective 

^enwnstra^ -The materials developer — 

reviews these translations. The project director settles 
disagreements between the two developers. 

6. The director and the two developers decide how the teaching 
of the objectives should be sequenced in the materials (e.g., 
easier objectives should be taught earlier.) Objectives 
should be deleted which are not 

a. Considered to be important by the project team or the 
Mastery Learning Panel, or 

b. Perceived as desirable/essential for developing mastery 
in other objectives in the course, or 

c. Perceived as desirable/essential for learning other 
competencies, knowledges, and/or skills outside the 
course . 

7. The test developer estimates how much student time is re- 
quired for testing each objective and adds them all up to 
get an estimate of total testing time for the whole course, 
If this is too great (compared to the time students should 
spend learning from the text), then enough of the least im- 
portant objectives should be deleted to reduce the total 
testing time down to a reasonable level. The resulting re- 
duced list of objectives constitutes the test blueprint . 

8. The test blueprint is reviewed by the Mastery Learning 
Panel for the comprehensiveness and fidelity of its repre- 
sentation or "mirror 11 of the course, and recommends modifi- 
cations. The Project Team reviews the recommendations of 
the Panel and, if needed, adds or deletes descriptive ob- 
jectives, and revises the sequencing of the objectives. 
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See the Appendix to this section to see how the Blueprint 
is related to objectives and domain specifications. 
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Develop Domain Specifications . This step expands the de- 
scriptive objectives into domain specifications (DS) . Each 
of the test blueprint's objectives must be included in at. 
least one domain specification (DS) , but each of them can 
be included in more than one DS, and a DS can include more 
than one objective-. An objective can contain both knowl- 
edges (amenable to paper -and -pencil testing) and skills 
(amenable to observational appraisals of students using 
and applying the knowledge). OS's should be spelled out 
for both paper -and- pencil and application or performance- 
based items. If the DS is for performance -based testing, 
then the physical and social context for the testing, tech- 
niques and devices for scoring, and materials and perfor- 
mance aides needed for the test .should also be spelled out 
in the DS. (See the examples of such a DS at the end of 
this section in the appendix.) . 

The test developer reorganizes the.blueprin 's objectives 
in terms of test format and environment (paper-and-pencil 
vs. performance-based), thematic relatedness among objec- 
tives, and other factors (personnel requirements, methods 
of scoring, etc.). The test developer then develops out- 
lines of specifications of all the blueprint's objectives 
and develops complete and detailed specifications of three 
typical OS's; two sample test items are created for each 
of the three complete OS's. 

The director, materials developer, and Mastery Learning 
Panel review both the reorganization of the blueprint and 
the test developer's methods for developing complete OS's. 
They recommend guidelines for the test developer's further 
construction of OS's for the other domains. Particular 
attention is devoted to developing general techniques for 
increasing the appropriateness of the sample test items as 
indicators of the competencies defined by the OS's. 
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12, Estimate average test length.* (This task raises serious 
technical issues that lie at the heart of the development 
of CRT's.) The test-length problem is intertwined with two 
other issues: 

• How to estimate student domain scores; and 

• How to sort students into mastery and non-mastery 
groups. 

To get a general idea of the technical issues that are at 
stake , I here quote some passages from Hambleton's** recent 
overview of the state-of-the-art in CRT development: 

'*The length of a [CRT] (...the number of test items measuring 
each objective in a test) is directly related to the usefulness 
of the [CRT] scores obtained from the test. Short tests pro- 
duce imprecise domain score estimates and lead to mastery de- 
cisions that prove to be inconsistent across parallel -form or 
retest administrations. Therefore, CRT scores obtained from 
short tests have limited value... When CRT's are us6d to assign 
examinees to mastery states , the problem of determining test 
length is related to the number of classification errors one is 
willing to tolerate. 11 
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' "Estimation of Examinee Domain Scores 

Several methods are available for the estimation of examinee 
domain scores. •• [The simplest estimates is the examinee's pro- 
portion-correct score]. This estimate is highly unreliable when 
the number of items on which the estimate is based is small. 
For this reason, procedures that take into account other avail - 
able information in order to produce improved estimated ... would 
be desirable. [One method utilizes] information gained from 

* 

For more on this and on how frequently tests should be given, 

see Appendix. 
** 

Ronald K. Hambleton and Hariharan Swaiminathan, "Criterion- 
Referenced Testing and Measurement: A Review of Technical Issues 
and Developments" in Review of Educational Research , Winter 1978, 
Vol. 48 9 No. 1, pp. 1-47. (All page references are to this 
article.) 
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the group to which an examinee belongs [the information is the 
mean, score of the whole group] • [This method] may not be ideal 
since it does not take into account any prior information that 
may be available... A Bayesian solution.., takes into account 
not on ly di r ect a nd~ colla ter al info r ma t ion but a isxr prior infor- 
mation... Direct information is provided by the examinee f s 
test score; collateral information is contained in the mean 
score of the examinee f s group; prior information on an examinee 
may come from past test performance or the examinee f s performance 
on other objectives measured by the [CRT]... The results of this 
study bear out the expectation that Bayesian estimation proced- 
ures are the most efficient in the estimation of domain scores. ff 
(pp. 5-7, 9). 

" Allocation of Examinees to Mastery States 

A second major use of scores obtained from criterion- 
referenced tests is in the assigning of examinees to mastery 
states or categories. It may appear tempting to first estimate 
an examinee's domain score, compare it with one or more cut-off 
scores defined on the domain score scale, and then (for example 
in the case of two categories) classify the examinee as either 
a master or a nonmaster. Typically, this is the strategy 
adopted by individuals implementing obj ectivesrbased instruc- 
tional programs. Unfortunately, this approach is not usualTy ;; 
very satisfactory. One reason is that users must assume all 
classificatipn errors (whether they be of the "false-positive" 
or "false-negative" type) to be equally serious. This is an 
unreasonable assumption to make in many instructional settings. 
For example, with instructional objectives that are prere- 
quisites to more advanced ones in a curriculum, false-positive 
errors (moving examinees ahead before they are ready) may be 
far more serious than false-negative errors (holding examinees 
back, even though they may have "mastered" the objectives in 
question) ." (pg. 10). 
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The Project team discusses how often teachers and studeats 
. will want student mastery to be tested as the student progresses, 
and how much time teachers and students can tolerate for testing, 
scoring, and interpreting a test given at a typical juncture in 
the course. The Team devises an "optimal" test duration. 

In our own bank checking mastery learning project, we de- 
cided to place tests at natural breaks in the content, for ex- 
ample, between the teaching of deposit slips and the teaching 
of check writing. We discussed various lengths of testing and 
made arbitrary decision that tests should take about 15 minutes. 
Perhaps final tests could be longer. We did not know how many 
items most students could complete in 15 minutes, but we guessed 
that it would be good to limit the tests to 10 or less items. 

A. 

We recognized the problems this would cause with reliability, 
but we chose to sacrifice reliability for the sake of making 
them easy to use . 

The Team then sketches both the instructional and the tech - 
nical pros and cons of the test "optimal" duration. The team 
compares and weighs the trade-offs of the test durations teachers 
and students could tolerate versus the sacrifices in technical 
validity and reliability imposed by that test duration. (The 
basic trade-off is: shorter total testing time requires either 
(1) fewer tests (less frequent testing), or (2) testing fewer 
objectives, or (3) risking larger errors in classifying students 
as masters and non-masters . 

The final decisions about chunking instruction into units 
or lessons to be tested should now be made. If your instruc- 
tional material already exists, tests should test whole lessons 
or groups of whole lessons. (For a poorly designed lesson that 
should be split up, your test on that lesson will just have to 
be either too long. or too selective in the objectives it tests.) 
If you are developing the instructional material simultaneously 
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with the test development, then chuncking the instruction should 
involve considerations of test length and how often tests are 
given but should not be solely determined by such considerations. 

This final chunking or segmenting should be displayed in the 
Test Blueprint. When the latter displays this segmented testing 
plan, we call it the Segmented Test Blueprint. You are now 
ready to develop the actual tests . 

13. From the segmented blueprint, the test developer drafts 

test items in quantities designated by the Step 12 decision 
regarding test length. For each objective requiring a per- 
formance- based simulation format, the test developer speci- 
fies the context, performance aids and other resources 
required by the objective's domain specification. The 
Team as a whole examines >^nd edits all the items, and 
develops a draft of the test's directions to students and 
directions to administrators of the test. 



In our letter to potential teachers at pilot- test sites, 
we attempted to explain what skills students needed to be able 
to do our checking kit materials (read at a third grade level, 
arid add and subtract 5-digit numbers). We decided against 
using a placement test. Instead we trusted in the teacher's 
evaluation of the student 1 s ability. This didn't always work. 

We put the directions to the administrators of the tests 
"xn the Teacher's Guide. They were very simple, since directions 
given to students contained all that was necessary for taking 
the test. Teachers were told verbally that tf^ey could explain 
what the test was asking for if students had difficulty under- 
standing directions or parts of an item. We asked teachers to 
note such sections that caused difficulty and let us know about 
them. 

14. Prepare a scoring method . If the selections of objectives, 
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domain specifications, and test items have been carried out 
correctly, developing a scoring method for paper-and-pencil 
items is straightforward. Developing scoring methods for 
performance-based items is more difficult but feasible. 
The Evaluator and Director collaborate in designing and 
formulating scoring methods that can be adequately utilized 
by minimally trained teachers and paraprofessionals without 
sacrificing their yield of valid information about student 
mastery. 

One teacher used our scoring instruments religiously. 
This gave her the advantage of being selective in the re- 
teaching materials to which she directed students. On some 
of the tests, the teacher's score key directed students to 
specific materials related only to the test item that the 
student got wrong. Because tl 's teacher carefully used the 
score key, she was able to devise more efficient learning 
paths for her students, i.e., students went only to re- 
teachings specifically related to items they got wrong. 
The other teachers that we observed simply sent students 
back to extensive reteaching that covered all the content 
related to the whole test. This slowed student progress 
and reduced their motivation. 

Our decisions about scoring method were< guided mainly 
by considerations of simplicity. Even so, the scoring 
keys we developed were too complicated for most teachers 
to use. For example, on the check writing test we had 
teachers checking each section of the test on a list. 
Most teachers ignored our check list and simply compared 
the bank checks to correctly filled-out checks displayed 
in the answer key, and marked or directed student f s atten- 
tion to places where errors occurred. 

15. The final technical challenge here is to decide mastery- 
level cut-off scores for individual tests. How many items 
in a test of an objective must the student answer correctly 
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to be justly considered a "true master" of that objective? 
Educational researchers and psychologists have not resolved 
this yet, so you will have to use your own good judgement. 
Nevertheless, here are things we found helpful to keep in 
mind: 

a. Educational consequences : if mastering this objective 
is essential or important to learning later objectives 
in the course (or text), then make the cut-off score- 
(below which tested students are considered non-masters) 
higher and more aemanding (perhaps 80t-100t correct} 
than you would have made it if the objective did not 

| 

have those educational consequences. 

j 

b. Real-world consequences : if not mastering this objec-. 
tive could result in the student's embarrassment or 
failure to achieve a 'practical goal he or she has , then 
make the cut-off score higher. 

c~ Reteaching consequences : don't make the cut-off score 
so high that students who repeatedly fail the test will 
not be able to relearn the material adequately because 
not enough reteaching materials* are available. Higher 
cut-off scores mean that more students will not pass 
the test the .first time around, and that students who 
don't pass are more likely to not pass the second and - 
third times around. 

d. We based our cut-off .score decisions on our perceptions 
of how crucial a given skill was to success in the real 
world. But we made sure that the perceptions of many 
different judges were consulted to ensure that indi- 
vidual biases and prejudices would cancel out. The 
more subjective a problem is, the more critical it is 
for you to involve as many judges as possible to re- 
solve the problem. For subjective problems, "objec- 
tivity" means "intersubjective agreement." 
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The Mastery Learning Panel reviews the draft of the tests 
examinihg 

a. Its items for appropriateness to student ability level 
appropriateness to the content, and fidelity to the 
original objectives. 

b. Its directions to students and directions to test ad- 
ministrators for clarity and usability by intended 
users. 

c. Its scoring procedure for clarity, simplicity, and 
fidelity to the objectives and to the domain specifi- 
cations. 

d. The frequency of the testing and the lengths of indi- 
vidual tests. 

The Project Team revises the draft accordingly. 
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APPENDIX 

Preparing Instructional Objectives : 

1. An objective is a collection of words, symbols, and/or 
pictures describing one of your important intents. 

2. An objective will communicate your intent to the degree 
you describe what the learner will be DOING when demonstra- 
ting achievement of the . objective , the- important conditions 
of the doing, and the criterion by which achievement will be 
judged. 

3. To prepare a useful objective , continue to modify a draft 
UPtil these questions are answered: 

* „ * 

. What do I. want students to be able to do? 

What 'ate the important conditions or constraints under 

which I want them to perform?'. 

How well must students perform, for me to be satisfied? 

4. Writt a separate "statement for each important outcome or 
intent. Write as many as you need to communicate your 
ihtents. r . 

* * ■> * 

5. if you give your written objectives to your students, you 
may not have to do much else. 

' Why? Because o-ften students are already able to do what 
you are .asking them to do and will b-s happy to demonstrate 
their ability, now that they inow, what is wanted of them. 
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Important questions to ask while writing objectives: 

• - 

1. What is the main intent of the objective? 

2. What does the learner have. to do to demonstrate? achieve- 
ment of the objective? . * . 



/ 

3. What will the learner have to do it with or to? And what, 
if anything, will the learner have to do without? 

4. How will we know when the performance is good enough to 
be considered acceptable? 

Developing Domains : 

One strategy is to start with prototype items (perhaps 
extracted from teaching materials) and systematically alter 
parts of those items to generate sets of equivalent items exem- 
plifying the general concepts or skills supposed to be tapped 
by the pr ;otype. Next task: determine whether satisfactory 
performance on each of these prototype items is really an anti- 
cipated outcome of the instruction and determine to what extent 
the list of items may represent a biased subset of some larger 
set of implicit outcomes. To do this, ask the writer: M Is 
this what you would like students to be able to do? Is there 
anything else you would like them to be able to do? In what 
ways might the items on this list be changed and still get at 
the sanje general ideas? 19 This line of questioning produces 
sets of items that the writer can classify in te.ms of general 
descriptive titles. The resulting pool of items may be large 
or small depending on the subject matter. It may consist of a 
sample listing of items or it may be composed by starting sets 
of rules by which items may be generated. In general, the task 
is to produce a pool that the evaluator and writer agree upon 
as representing the central body of knowledge and skill that 
are the goals of instruction. 

Writing Objectives 

If you are developing mastery tests for existing materials, 
it is best to formulate objectives with the author or with 
teacher ; who are most familiar with the materials to help the 
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test developer decide what the instructional intent was of 
each chapter or section. After they explain what they were 
v trying to teach on each subject, the test developer writes the 
learning objectives in ways that can be useful for testing 
purposes. 

In developing objectives for our kit, we decided that none 
of the objectives ask students to discover knowledge, but to 
ask students questions about knowledge that was taught in the 
materials. A few of the objectives called for students to use 
knowledge they had gained, e.g. write a check (as they were 
taught to write checks). Most of the objectives called for 
students to remember knowledge either by recalling , e.g. fill 
in the blanks, or by recognizing , e.g., matching/multiple 
choice. 

We decided that all crucial skills, such as filling deposit 
slip, check writing, reconciling, and endorsing must be learned 
and tested at the use level.) Learning objectives covering these 
knowledges called for students to demonstrate ability to use 
that knowledge. 

I 
I 

Other knowledges and skills -- such as knowing what a can- 
celled check is, knowing how \ to choose a checking acount 
could be learned at the remembering level of learning. For 
example, students were asked to remember factors involved in 
choosing a checking account. And to make it easier, we asked 
them to recognize rather than recall those factors. 

\ 

! Our first list of objectives consisted of 110 stand-alone 
objectives. This was confusing to the Mastery Learning Panel. 
So we decided to divide objectives into groups of objectives 
that would be included on given tests. For example, if we saw 
need for dividing the existing materials into 10 sections and 
we planned on having a test for eac^i section, then we would 
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have 10 clumps of objectives. The Mastery Learning Panel was 
then ablfc to evaluate and rank objectives in each clump, thus 
helping us de ve l°P ers to weight each test. (By weighting we 
mean determining which objectives in given tests are more im- 
portant and should take up larger amount of the test.) The 
Panel was also able to indicate which clumps of objectives 
were more important and which deserved minimal or no testing. 



Examples of objectives we ised for our kit: 
Test # Objectives 

1. Given five blank checks, student will be able to correctly 
write two checks including: (1) date, (2) check number, 
(3) payee, (4) dollar amount in numbers, (5) dollar amount 
in words, and (6) signature. 

2. Student will be able to correctly complete a deposit slip 
including on the slip: (1) the date, (2) amounts of up to 
three checks totalling n0t more than $500, (3) a cash 
amount, and (4) a total amount. 

3. a. Student will be able to correctly endorse a check 

intended for deposit only. 

b. Student will be able to endorse a check for cash. 

c. Student will be able to correctly write a check for 
cash using his own name as the payee. 

4. Student will be able to correctly write the following 
items on a check register: (1) check number (if check), 
(2) date, (3) description of transaction, (4) amount of 
payment or deposit, and (5) balance. 

5. a. Student will be able to recognize the process of open- 

ing a checking account including: 

- going to new accounts desk at bank; 
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- selecting kind of account; 

- signing card; 

- depositing money; 

- receiving checks. 

b. Student will be able to recognize the correct procedure 
for signing the new account card and signing subsequent 
checks. 

c. Student will be able to recognize definition of 
"endorsing". 

jd. Student will be able to recall two forms of identifi- 
cation frequently used when cashing checks. 

e. Student will be able to recognize tjie definition of 
"cancelled check". 

f. Student will be able to recognize the definition of 
"overdraft". 

g. Student will be able to recognize what he must do in 
case he receives notification that he has overdrawn 
his account. 

h. Student will be able to recognize why it is important 
to keep an accurate current balance of a checking 
account. 

i. Student will be able to recognize the steps to take 
if he makes a mistake on a check, including: 

- write void on check stub or check register; 

- write a new check. 

j. Student will be able to recognize why it is important 
to notify the bank if any checks are missing. 
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TEST DEVELOPMENT PROCESS 

o 

Actual example from our process for developing two test items 
for our Mastering Checking Kit^ . 

I. Start with the "Test Blueprint 11 cf all the tests for the 
Kit. (Our "Test Blueprint" outlined 5 tests for each of 8 
lessons ■ 40 tests. Below is the third test for Lesson #2 
on "Depositing".) 

Test 003 

1. [251] Student understands the purpose of the deposit slip. 
(Knowledge test item (paper/pencil) required (see page ).) 

2. [501] Student correctly completes z deposit slip including 
on the slip: 

- the date; 

- amounts of up to three checks totalling not 
more than $500; 

- a cash amount; 

- a total amount. 

(Performance test item (full domain specification is 
required) .) 

II. Develop domain specification: 
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•miii incmcmof 



Oejeetlvo 

Staleat etl\ aa aale to correctlv eoaplata a Aapaalt allp 

ueuTut taa .art in *• Aatc. •« ■» £ 

tint caecta tetalllM aot mt taaa $5§0 # I3f • 
umt « ul <•> • total aaaaat. 

Coataet/toaatler Oeaala 

1. StaAeat via aa atpoctrt to (ill aat toe correct 
Aepealt aUpa. 

a. If tee ataaaat aaa eat Aeoeoctratrt aaatarr after 
atteeptlea Clva Aepeoit all pa. to aiaali 
AlaceatUea teetlaa art ce to taa aparapclata 
re-toacitef atrateff. »t taa Itacwt H af tka 
taterAeecaer. a ataAeet atfet ft* flaee taa 
tlUtUui Aaaaait all pa ta C1U aat laatert af 
aalaf AlrecteA ta a re-taecelee atiataej. 

u tt«. a ataaaat aaa fllUA aat taa aaaaait ellee 
ofclek aT Caala are correct, ee .111 after tftee ta 
Ua tatarAaacaae Car aealeatlee,. 

a. Ciecta ta aa lecleArt aa Aepeoit allpa .111 ee 
frea fletUlaaa paraaaa ar fletltleea ceeecalee. 

$. StaAeot alll aa 41 racta* ta aaa taa correat Aata aa 
taa Aepeoit allpa. 

5. 1 eaaeec rapraaaatlaf a fletltlaeo caei aeoeat 
alll aa eiaaa ta taa ataaaat far laclaalaa aa taa 
Aaaaait alia. Tele aaaaat elll fte a Aollara art 
eaata aaaaat aat creator taaa HW. . 

T. ataaaat aia aa olvee oaff leleet lafaraatlaa far 
taa arttlaf af taa Aepealt allpa* 

t. ft t leeet aaa af taa three caeca aaoaata alll aa 
for a aaa af aaaaf U aaaa A all era art rt toul aaa 
Ul aa far aa aaaaat la Aollara 



at bar ckact aUl 



taa tetal aa taa aapaalt alias alll aat atcart 
aaaa.aa. 

lUa 

1. Taa ataaaat alll aa flfaa flra ftlaat Aapaalt allp 
ferae* 

2. fitter aataaatle Aapaalt allpa tM»«lWT*£ 
•trteafa aaa aaaaait ellee) ar *»jV°~ d !!!!iS 
allpa alll aa aart. PecaAo-Aeeeelt allpa oaeeU 
appreileete raal aapaalt allpa aa cUaaly m 
paeolftle. aat oeeolA ee air taa aa aaapla Aaaaait 
allpa! PeeeAe-Aepaeit allpa aeeelA aa leoce fl.e. 
! .am pafTat Aapaalt allpa la aat accept. Jle). 
If aataaatle Aapaalt allpa ara aart. tft»r oeoalA aa 
AaotreToA leeeAlatolT fello.lae taa oaerclce. 

3. taa af a paa la aeceeearr. 

a. it taa Alecretlea af taa tatorAeaceer. taa 
atrteet alll aa allaart ta aaa laotroctloaal 
aaterlala far aaalataaea aalla flUlaf aat taa 
Aaaaait all pa. 

5. 1 elect placa af paper u alio art. 

a. Uaa allaart far arltlaf Aepeelt •*« ■{ 

taa Alaeratlaa af taa tatorAeecaer. aat alll 
aaacllf aat atcart flftaaa electee. 

7. Taa ftlaat Aaaaait aUpa alll aat fte attecert ta a 
caaek hart- 

A. Taa aat Iraaaaat aaaalA aa ceeteref aroort a aall 
llcatrt arltlaf eorfacc. 

a. Taa aaa af a calealator to aAA taa ceeci art caat 
aeeoate alll aa allaart at taa Alaeratlaa af taa 
teterAeaceer. 
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See pi a Street leee 



Selee are tee copcelt •lip etercloee. loa alll ec alvae 
flaa elaafe eepeelt *Um- Xf f*« aata a aletake, start eaer 
m eeettec blMl ceeoelt alia, taaa roe have eoaalatad taa 
aeeeelt eltee velca tee feel ara car r act, aaaa tfcaa ta ?eac 
teacher* if taa eopoelt ellee ara correct, foa bate 
probably ha* aaaaaa practice. If there ara fttm af taa 
lepeelt aliaa aa velca faa atlll aeec ta le prate, toar 
teacher aill pceHee aora araetlca. taa toeaya cate aa taa 
aepeett aliaa. 

U Sappoee faa aaat ta desalt taa follotlae, caoefce 
ia ?eee ceecklef acceaat: 

a choc* far f IS fraa Jeee lie tar 4 a. 

a parched far $110. S7. 

faa aiaa aaat ta ceeoelt SM.S7 la caee. 
nil cat a iaaaalt ellp ta aa tale* 

a. Sappoee yea aaat ta eepeelt taa folleala* caack a la 
faar eiecklae; accaaat 2 

a paycheck far 210. If. 

a refeet caack for S1J.M fraa a etere. 

a caack far SS5 fraa Aaaa Joeac. 

faa alee aaat ta depeclt fTUIS le caaa. 
rill aet a iaaaalt ell a to aa tale. 
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Scoriae gay 

Oat a U Ccrroct locatloe. 

lata correct. 

Ckvck a ao aata la corract locetloue. 

•• eck a so a ate correct. 

*• eeoeet correct. 

Caaa eeoeet le correct locatloe. 

7 * Total aeoaat le correct locatloa. 

•* Total aeeaet correct. 

*• __l«eaerltiae lcflble. 

taa ear e cleer. 



Steeeet le Meet If lee ae a eaeter af tale 
eklll If ell ef the criteria ara eerkee ae 
accept a ale. 



III. Develop Answer Key page for whomever scores the test item. 
The student receives directions (not shown) and two blank 
deposit slips. Teacher examines student's work and scores 
it according to the answer key below: 



Correctly f 11 lei out depoalt alipci 



Da pot It sllpe 
003 Score key 



I* K 



tie eo 
i4 yi. 



mm**** Mitt 



0 
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Score 



Skill 



Green Yellow Sloe Review 



1. 

a/ 

4. 

7.* 



.Dote correct.—— 
~Ceen eoMMint correct.- 
Check aoounta correct. 



jrotel waount correct.———-- 
JUl the above In correct location 
Junevrltlft* lef Ible— 

ire clear. ■■■■ — »' 



tmm . 17,35 


7-fe 


20-21 


2 


— 17,35 


7-fb 


20-21 


2 


— M.J5 


7-fh 


20-21 


2 


— - 17.15 


7-tt 


20-Jl 


2 


17.35 


1~U 


20-21 


2 



If ell Itaee are aarked a, eo U one of theaet 
for tat new eeelf noent (roe teacher) 



IS 

— 17-42 



10-11 ll-M 



for iteoa aerkeeTx* fO to one of the eectione to the rifht of the Item 
(Or fct new eeeifaoeflt free) teacher r . 
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Element #1 of the "Test Blueprint" two pages back identified a 
"knowledge" or cognitive test item (in contrast to the "per- 
formance" test item you have just looked at). We developed a 
simple true/false test item for element #1 and included it as 
a "practice" in the student f s lesson. Below is a reduced 
replica of this practice ( f P f ) together with a replica of the 
teacher f s answer key page for this particular test item. 



n nark taa traa suttMiti wit* a J. Hark UXmm 
ata taa m ta vitu aa P. 

taa comet aata aa a oapoait slip tails i*m yoa Mdt 
tlM 4tf it. 

laa if— it slip •aoala' tall taa aaafc aaactly ahat you'ea 
dapaaitlaf. 

If taa taller ai a o o aata taa eaaa yoa aapoait, you eait do 
aotaiaf aaaat it. 

It's aot laportaat to fill oat aaoaata oorracUy an your 
Japaait slip. laa tallar will aa* it all ap for yoa 
aayaay. 



\ jap^a t a t a w a f wick a X* fal— 
T lao aamtt aata « a aajaatt slip talis aai ym 



T ^aajaait^aMp^ala^M tall taa taafc aaatUy < 
F If tat tallar ■Imaii taa aa yaa aajaiit, : 



Ftt'i aat aajwi ta fill aat mtmtt mnmtiy 
alia. Tkm tallar will mm tt all 



Items were also developed for elements #3, #4, and #5 in the 
"Test Blueprint 11 section on depositing. All three were of the 
"performance 11 type. 

7r, 
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Frequency of testing/fear of testing 

Early in our project, we wondered about students' attitude 
toward tests. Contrary to fears, we discovered that very few 
students resented the tests. Most looked on them as chances 
to prove they were learning. Some appeared to like the tests 
and rushed through the materials so they could get to their 
next tests and prove they could do what they had studied. 

Our kit requires that teachers give thirteen tests. Fre- 
quency of testing is determined by subjects covered. Since 
easier concepts and skills are dealt with earlier, the students 
take tests more frequently near the beginning and are more likely 
to pass on the first try near the beginning. This keeps the 
teacher busy administering testing at the beginning. As skills 
get harder, the interval between tests lengthens and the teacher 
can breathe a little easier and give greater attention to stu- 
dents as they go through the teaching materials. 

We wondered whether we should relieve the teacher by group- 
ing more skills together in the earlier stage, so that those 
tests would cover more teaching material and therefore come 
less frequently. This presented two problems: 1. It in- 
creased the chance of student failure (the student's early 
experiences of being tested should be success experiences), 
and 2. The subject matter in the early lessons did not lend 
itself to being grouped together. 

One solution: teachers make copies of half of the answer 
keys available to students and let them correct half of their 
own practices and tests in the early lessons. This alleviates 
the load of correcting many tests and practices for the teacher 
and teaches students to take responsibility without depriving 
the teacher of ultimate control. 
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Length of tests 

Early tests can be brief and relatively easier to ease 
students into the materials. In the field- test of our kit, 
success at doing tests was a delightful experience for most 
students, especially if they were used to failing. If they 
did fail, the tests were small tests; they did not have to 
invest much in them and they did not have to cover large 
amounts of reteaching material to prepare for the next test 
on the topics they didn't master. It's no big deal to do a 
three or four question test over again. 

Later tests can be more lengthy. None, though, should take 
over 30 minutes. In the most important tests, the students 
should actually perform functions they'll have to do as masters 
of the course's material. This is not busy work, and they get 
a chance to experience being real masters when they complete 
these more difficult tests. 

COMMENTS ON STAGE THREE: Model Tests, Reteaching Lessons 

Te aching Plans 

Adapting the plan to current resources and needs 

Stages One and Two sometimes cover a considerable span 
of time. Changes occur that require modifications of the 
earlier plan. Newly hired staff should study the plan carefully 
at the outset of Stage Three and express their ideas about: 

1. Modifying goals (within limits that a person will allow). 

2. Modifying procedures for reaching goals. 

3. Modifying staff assignments (who does what). 

4. Modifying schedules; i.e., 

a. Changing" the order of events 

b. Changing deadlines. 

c. Developing short and medium range schedules (time lines). 
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To gain support by all members in the project team, give 
all an opportunity to participate in shaping the plan and de- 
fining their roles. Hold them to the major intent of the project, 
but challenge them to improve the proposal design and set 
realistic limits on what can be accomplished in light of: 

• Their abilities, experience and interests. 

• The project f s resources (facilities, funds available). 

• Needs perhaps the sponsor wants to meet needs other 

than those addressed in the original plan. 
Needs envisioned in Stage One or Two may have 
changed. Or the ability to meet those needs 
may haye to be re-evaluated. .Or project staff 
may want to convince the Sponsor to add other 
needs or replace needs addressed in the plan 
with alternate needs. 



Final decision on content of instruction 

Plans in Stages One and Two suggested that the mastery 
learning material to be developed would cover certain subjects. 
In light of the time, resources and staff available, were these 
goals: 

• Too broad? 

• Too narrow? 

• Misdirected? 

Decide how much content you can realistically cover (e.g., 
one aspect of money management rather than the whole field of 
money management) • 

Decide what specific areas of content your team is most 
able to cover, "V.g., the team has talent or experience in 
teaching banking skills but not in budgeting. This decision 
must be made in light of: 
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• Project staffs interests and abilities. 

• Needs of the audience. 

• How easy it is to do one content area as opposed 
to another, e.g., it's easier to design measurable 
objectives for banking skills than for budgeting 
skills. 

• Availability of good teaching materials in one sub- 
ject area but not in another. 

\ If you are not sure how much content you can cover, choose 
one of the easier content areas and plan to develop it through 
t^e pilot test stage. Experience in developing the first con- 
tent area can guide you in deciding how much more content 
you'll be able to cover. It will also help you avoid mistakes 
made on your first efforts, so that development of other con- 
tent areas will be more efficient and less expensive. 

\ 

i 

Define instructional intent 

ftace the content to be covered is decided, then decide 
what's 1 most important to teach within that content area. Ask: 

•\ What level of competency should the student attain in 
this content area. 

^L. In a range from bare survival competency to 
expertise in all details, where do we want the 
students to be? ^ 

2. In what context do we want students to be able to 
demonstrate mastery? 

• What skills and knowledge are absolutely essential for 
the level of competency we want the students to attain? 
e.g. in the area ot bank checking, must they know how to 
fill out both checks stubs and registers, or is jthe 
ability to fill out check registers sufficient? 
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• What level of knowledge should they attain on each 
essential skill or knowledge? For instate, in the 
area of bank checking, 

1. Usinfr skills : Should they be able to write checks 
correctly giving a variety of payees, amounts, etc? 

2. Recalling information : Should they be able to 
recall the purpose of a deposit slip? 

3. Recognizing information : Given a list of items, 
should they be able to select the items that are 
important factors in choosing a checking account? 

Preliminary selection of important objectives 

Consider who should decide key issues in defining instruc- 
tional intent, «and in writing and prioritizing learning objec- 
tives. Examples: 

a. Project staff. 

b. Representatives of the sponsor 

c. Publisher's representatives. 

d. Representatives of students, teachers, and/or educa- 
,y tional programs or institutions who will use your 

materials. 

If the sponsor or eventual users have the power to kill 
the project because of disagreements over content covered or 
learning objectives, then they should be consulted in the mak- 
ing of decisions about content, instructional intent and ob- 
jectives. Certainly these key people should have some input * 
at this stage of selecting most important objectives. Consider 
whether to convene r committee of representatives from the group 



or to get sep 
groups. 




responses from representatives from these 
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Questions to be asked at *t his 'stage; 

* . ' '' t : '; 

a. Which objectives are broad? • w - * . . 

»'»'.*• ' * 

b. Which objectives can be subsumed under ;broad c " - 
objectives? * ' 

c. Can any of the broad objectives be eliminated? 

d. Can any objectives subsumed under broad objectives 

be cut? ' • * *'.•••,'• • 

These questions need to be asked in light of . publisher's \\ 
teachers (buyer or gatekeeper.) interest as well as student's > 
need. 

Make Test Blueprint 

The test developer will want: ^ ^ - 

1. Enough items per test to make it possible to evalu- 
ate that test and its items for reliability. > * 

2. Tests to contain items thatt demand the same kind* 
of knowledge ( using skills in one testers, items . 
that ask for recall of information on another test) . * * 

These wants may conflict with the way the teaching materials 
should be presented. What if a chapter presents only a few ob- 
jectives? Should a disproportionate number of items be written 
to test these objectives to satisfy want #1 above? Or should 
the test for this chapter be loaded onto a test for an adjacent 
chapter? ^ 

What if a chapter contains objectives on two or three 
kinds of competence? For example, a check writing chapter could 
test competence on the use level (write a check), recall level 
(state the reason for always signing your name the same way), and 
recognition level (the circled numbers on this check are: a. the 
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account number or b. the check number). 

Even though it's awkward to test all of these kinds of compe- 
tence on the same test, should the content of the chapter dic- 
tate the test? Or should you make more than one test for the 
chapter? Or should you gather miscellaneous objectives from 
the same kind of competence from a number of chapters and put 
them in the same test? 

(In Masterin g Checking , we had a number of recognition 
level objectives scattered across five chapters which stressed 
use level objectives. Rather than mixing kinds of competence, 
we kept the tests for those chapters totally on the use level. 
Then, in a "review" test, we tested all the recognition level 
objectives. This forced us to add a review chapter in our 
teaching materials to go along with the "review" test .) 

Conflict may arise between test developer and the ma- 
terials developer. The latter will want chapter tests that 
cover all the objectives in a chapter and only those objec- 
tives. Or, if two or more chapters are to be tested by one 
test, then the materials developer will say ell the objectives 
of those chapters should be tested. 

The test; developer won't be as sensitive to the logic 
of the content presentation. The materials developer won't 
>e as sensitive as ttef test developer to the handicaps of 
chapters with few objectives to be tested or with objectives 
to be tested for different kinds of competence. Each of these 
persons should sensitize the other to his/her point of view. 

Decide, where the breaks should be 

This relates to the Test Blueprint, but it deserves 
special attention. 
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The blueprint-making process can reveal illogical 
structuring of already-developed teaching materials. For 
instance , it may seem more logical to have a test break midway 
in Chapter One and have another test that covers both the 
last half of Chapter One and all of Chapter Two. But this will 
cause problems. Instead, the test and materials developers 
should work toward making all chapter breaks coincide with test 
breaks. 

Our nilot test observations indicate that tests that come 
in the miudle of chapters cause tMse problems: 

1. Students read on to the end of the chapter before 
taking the test. They bite off more information 
than they can handle. The information overload 
has a negative impact on their test results. 

2. Students don f t know where to begin in the teaching 
materials after completing the test. 

, Evaluating the Test Blueprint 

The Mastery Learning Panel should evaluate and revise the 
blueprint. The Panel should include: 

* Authors of existing materials and materials yet to be 
developed. The former will check to see if the blueprint 
affords valid testing of what has already been written, 
the latter have the same concern for materials yet to 
come • 

* Representatives of the sponsor or publisher of the kit. 

* Representatives of eventual users of the kit. At this 
stage, you may have to deal with people who aren't 
actual students or teachers, but vhose experience allows 
them to speak for the end users. 
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The purposes of this committee are to: 

1. Judge whether the test blueprint (outline of all the 
tests to be used in the kit) adequately covers all 
the previously agreed upon objectives that the kit 
is supposed to help leai-srs achieve. 

2. Approve of any changes in learning objectives that 
came during the process of writing the test blueprint 

3. Suggest changes in the blueprint. 



Developing the model reteaching lessons 

Write one or two lessons exhibiting the instructional de- 
igns you intend to use. Do the main elements of these lessons 
in as many alternative variations as you plan to eventually use 

Example: A model lesson on using a check register. 

Approach A: Use cartoon characters that show how to 
fill out check register forms. Write 
instructions in balloons. 

Approach B: Use numbered instructions and arrows from th 
instructions to the parts in the check 
register which the instructions are about. 

Approach A: Break the practices down into sub- skills, 
i.e., recognizing column headings (date, 
check #, etc*) one at a time. Filling 
out a series of check number columns, then 
filling out dates for each of those checks, 
then writing description of transaction for 
each check, etc. 

Approach B: Students learn the parts by doing the whole. 
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Doing one check at a time, they record 
the whole record for that check, going 
across all the columns each time. 

The above explanation of approaches A and B shows the 
kind of thinking that has to go into development of reteaching 
materials. 

Approach A uses more motivational gimmicks, cartoons, 
jokes, amusing names on forms, and amusing practice items. This 
approach is for reluctant readers. It also breaks acquisition 
of skill/knowledge into small parts, providing practice on learn- 
ing each part, then integrating the parts into a whole. 

Approach B is leaner, more straightforward. No cartoons. 
Illustrations are used more for informative than motivational 
reasons. Instruction relies more on words than pictures to 
convey information. Subheadings, titles, numbered lists and 
diagrams help readers quickly review each topic. Cues and 
other study aids tell learners where the main idea, examples, 
and practice are on each page. In Approach B, each lesson 
starts with an overview of the topic plus an epitomized or 
paradigmatic example displaying the highlights of what is to be 
learned. Then it teaches various parts of the topic, relating 
these to the whole as they are learned. As parts are learned 
they fill in details of the whole that was initially seen in 
broad strokes. 

« 

Developing the Model Teaching Plan 

The design of the tests and learning materials affects 
how the teacher teaches. For example, if there are no directions 
in the materials, telling students how, where, and when to begin 
and continue, then the teacher wifl be asked to provide thes$ 
directions*. - 
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In a class that's typical of your target audience, try 
out the alternative sample materials you develop. Have the 
teacher use them with a student, Otr „rve the tasks the teacher 
must perform to get the student successfully through the ma- 
terials. Based on your observations either: 

1. Revise the student materials to include directions or 
other means for guiding the student to do what the 
teacher would otherwise have to do, 

or 

2. Write a teacher's guide which shows the teacher what 
he or she must do to use the materials effectively, 

REMEMBER - 

You will probably do some of both of these. When develop- 
ing materials, you must think carefully about the demands the 
materials will make on the teachers. This is as important as 
knowing the demands the materials will make on students. If 
teachers find the materials hard to manage, no matter how good 

they are for students, they won't be used, 

<> 

Decisions to make if you're building enhanced mastery learning 
around existing materials . 

If you already have a basic text or basic, series and 
simply want to build mastery learning around whit you have, 
here .ari some questions to explore, d 

Are our exiting materials too lean for n^pny in our 
audience? If so, the reteaching materials to be 
developed need to elaborate it so that siu4ents who 
cannot handle the lean approach will receive more 
explanation, illustration and -practice. What if you 
discover that the existing material is so lean that 
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you think it f s unfair to test students if they've 
studied only the existing material? If that's 
the case either- - 

a. For cases where you can revise, scrap the exist- 
ing material, incorporating its better parts in a 
new basic test; or 

b. For cases where you can't revise, add complementary 
elaborations in separate companion materials that 
must be used along with the base materials before 
one takes mastery tests. 

'2. Are our existing materials too elaborate » i.e., good 
for slow learners who need abundant examples and 
practice but boring to other learners in our audience? 
If so, then reteaching materials would need to take 
a leaner approach. 

What if you discover that the existing material 
has much unneeded padding? If you can revise, then 
delete the padding. If you cannot revise, then tell 
teachers in a teacher's guide how to route students 
around the unnecessary sections. 

-3 

3. Are our existing materials written at a f reading level 
that's too high for many in our audience? If a sub- 
stantial part of the audience can handle the reading 
level and all other factors about the existing ma- 
terials are positive, use the existing material as 
the Main Text. Then make the Reteaching materials 
easier to read. 

There are other evaluative questions you can ask of the 
existing materials. Basically what you want to know is whether-- 

* The existing materials should be scrapped because 
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they are ineffective and/or inefficient, 

* The existing materials are useable if revised, 

* The existing materials need no revisions, but 
should be complemented by reteaching materials 
that make up for deficiencies in the existing 
materials or offer variations to match varia- 
tions in the audience. 

If the existing materials can not be scrapped or even 
revised, and if they are very poor, scrap the project! Don f t 
try to make mastery learning into a pretty package to sell 
a dead horse. Mastery learning packages are costly to build 
and they don 9 t work if all the elements in the package are 
not effective and efficient. Bottom line: make sure that every 
unchangeable part of- the package is instructionally sound. 

If you are developing enhanced mastery learning around an 
unrevi sable text . . . 

Though Reteaching Text will use a different instructional 
approach to present the content, do not alter the order of the 
reteaching chapters. For instance, in a text on bank checking, 
you might think the chapter on "choosing a checking account is 
more abstract and should be near the end rather than near the 
beginning. But, if it's near the beginning in the Main Text, 
keep it in the same location in the Reteaching Text. 

Students and teachers become confused if the topics of 
chapters in the Main Text and Reteaching Text are not parallel. 
Guiding students and teachers through the materials is much 
easier if chapters one, two, three, etc. in the Main Text cover 
the same topics respectively as chapters one, two, three, etc. 
in the reteaching Text materials. 
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The order of presentation within a reteaching chapter 
may be different from the order of presentation of that same 
chapter in the Main Text. But each chapter should cover the 
same topics. 



Obtaining exemplary mastery learning kits 

How do you find out about mastery learning materials? 

a. Search ERIC, as well as indexes of educational 
journals, for articles on mastery learning. 

b. Search ads for educational products in popular 
magazines, such as Learning Magazines , that go 
to teachers and school administrators. 

c. Telephone or write to key figures in mastery 
learning and ask them to tell you about latest 
innovative mastery learning programs and products. 

For a model of how to approach these key figures see my 
phone interview sheet: 

Key figures in mastery learning include: 

1. Benjamin Bloom 
University of Chicago 
Chicago Illinois 

2. Charles Reigebuth 

Instructional Design, Development § Evaluation 
Syracuse University 
Syracuse, NY 13210 

3. David Merrill 

University of Southern California 
Alta Loraa, CA 91701 
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4. Michael Katims 
Mastery Learning Development for 
Chicago public schools* 

5. Ronald Hambleton 

d. Call or write to publishers of mastery learning 
materials. 

1. ) Order catalogues that describe what they've 
done. 

2. ) Order samples of what they've produced. 

3. ) Ask for addresses of schools nearest to you 
that are using their materials. 

These publishers include: 

(Producers of 
Chicago Mastery 
Learning Program,) 

(Produces Benjamin ; 
Bloom- inspired High 
Intensity Learning 
System.) 
(Produces an 
elaborate mastery 
learning center for 
schools. ) 

(Produces materials 
using Merrill's § 
Reigebuth's kind 
of instructional 
design.) 
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1. ) Mastery Education Corporation 

85 Main Street 
Watertown, Mass. 02172 
(617) 926-0329 

2. ) Random House School Division 

201 East 50th Street 
New York, New York 10022 

3. ) Westinghouse Learning Corp. 

1500 First Avenue North 
Coralville, Iowa 52241 

4. ) Courseware Incorporated , 

10075 Carroll Canyon Road 
San Diego, CA 92131 
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Telephone interview format for eliciting infor mation about 
other mastery learning materials 

I'm from . We conduct programs [and produce 

materials] for adults and older youth who are illiterate or 
functionally illiterate. 

Currently I'm directing a project to develop a mastery 

learning kit covering the subject of • The kit is 

for adult and teen learners who are reading at third grade level. 

Here's what the kit will do for teachers: 

It will clearly portray the knowledges and skill communi- 
cated in the kit materials. 

It will provide criterion-referenced tests by which teachers 
can tell whether students have mastered knowledges and skills 

It will provide reteaching strategies teachers can use 
with those who have not raastwred. 

It will enable teachers to: 

1. Easily identify knowledges and skills taught by each 
unit in the materials. 

2. Test students to see if they have mastered those 
knowledges and skills. 

3. Reteach knowledges and skills not mastered. 
The Kit includes: 

1. Student profile sheets which list knowledges and 
skills taught by the imperials so that teacher can 
record mastery/non-mastery of students in each skill. 

2. Criterion-referenced tests which measure student 
mastery/non-mastery of skills and knowledges. 
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3. Re-teaching materials Cor strategies in a teacher's 
guide) and that can be used to help non-masters to 
achieve mastery. 

Would you be interested in seeing what we've produced thus far? 
Have you produced materials like this? How can I obtain it? 
Do you know others who have produced or are producing such 
materials? 

Their names, addresses, phone numbers. 

Reviewing state-of-the-art literature on mastery learning and 

on instructional design models appropriate for use in mastery 
learning. \ 

1 Ask experts such as Bloom and his associates, Merrill, 

and Reigeluth: 

1. Where can I read about mastery learning -- articles 
books? 

2. May I buy copies of your lecture notes? 

3. Do you have any detractors whose articles or books 
I should read? 

Search indices of educational journals for articles on 

mastery learning and instructional design, developments, and 
evaluation. 

t Here are some books/materials that may be helpful: 

Anderson, Lorln W. & Beau Fly Jones. Designing Instructional Strategies 

Which Facilitate Learning For Mastery , Paper submitted to AERA Conference, 
Junp, 1980. 

Block, James H«, ed. Mastery Learning, Theory and practice . New York: Holt, 
Rinehart, & Winston, 1971. 

Bloom, Benjamin S. "Learning for Mastery", UCLA Evaluation Comment > May 1968, 
vol. 1, no. 2. 

Godfrey, L. "Ten Steps to Managing Individualization", Teacher, Sept. 1977, 
O vol. ?5, 80-1. nrfc 
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Guflkey, Thomas R. & Judith A. Monsaas. 1 'Mastery Learning: A Model for 

Academic Success in Urban Junior Colleges", Research In Higher Education , 
1979, vol. 11, no. 3. 

Hambleton, Ronald K. & Hariharan Svaminathan, "Criterion-Referenced Testing 
and Measurement: A Review of Technical Issues and Developments 11 in 
Review of Educational Research , Winter 1978,' Vol. 48, no. 1, pp. 1-47. 

Hambleton, Ronald K. & Robert A. Simon, "Steps for Constructing Criterion- 
Referenced Tests", paper presented to AERA, Boston, 1980. 

Hassett, Mathew J. & Kathleen M. McCoy, "Task-analysis: A Post-Quiz Remedi- 
ation Strategy for Persorillized Self-Instruction", Journal of Education al 
Research , Sept/Oct. 1979, Vol. 73, no. 1. 

House, Peggy A., "Individualization", Science Teacher , January 1977, Vol. 44, 
pp. 20-1. 

Mueller, Daniel J. , "The Mastery Model and Some Alternative Models of Class- 
room Instruction and Evaluation: An Analysis", Educational Technolo gy, 
May 1973, Vol. 13, no. 5. 

Pencil, R.C. & G.J. Laabs, "Construction of a Criterion-Referenced Diagnostic 
Test for an Individualized Instruction Program", Journal of Applied 
Psychology , June 1979, Vol. 64, pp. 255-61. 

Popham, W.James, "Well-Crafted Criterion-Referenced Tests", Educational 
Leadership , Nov. 1978. 
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Identify best features of current mastery learning products 
and instructional design models 

Questions to ask as you evaluate mastery learning products 
2*nd instructional design models: 

1. Will this approach appeal to students/teachers in our 
audience? (Courseware's work so far is too advanced for 
slow learners in high school and ABE programs.) 



if 




Can our audience afford this kind of approach? 
(Westinghouse 1 s Mastery Learning looks very expensive.) 1 

Does our audience -have the resources required, forms, 
to back up the system. (The High Intensity Learning 
System requires a large library of resource materials to 
which students go for initial teaching and for reteaching.) 

What methods of presentation do 'these materials or 
models use that would ^e effective with our audience? 

What format designs would help us?, 6.gi, 

ways of packaging their materials 

ways of guiding teachers and students through 

the materials 
ways of. allowing for copying consummable portions. 
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Kinds of reteaching method and format options * 

Key questions to answer as you consider how to do 
reteaching: 

1. Should the reteaching come mainly from student 
materials? 

a. If so, should the materials be self-instruc- 
tional? 

b. If self-instructional, who will evaluate 
> progress the student or the teacher 

(i.e., will mastery tests be teacher- 
administered, ^evaluated and scored, or will 
f the student test and scoi'e himself)? 

I (Perhaps ABE and other adult training programs would 
; prefer that students be on their honor to test them- , 
selves and use answer keys to correct and score their 
"own tests. But most high school programs want tests 
• r- and answer keys under the control of teachers,) 

* c. If not self-instructional, to what degree will- 

the teacher manage the use of the student ma- 
terial? (Keep in mind that mastery learning 
advances masters or moves them to enrichment 
activities. Therefore students will inevitably 
be in different places in the materials. Hence 
teachers will find management extremely diffi- 
cult if the materials demand a great amount of 
teacher involvement.) 

d. Should the student materials be consummable or 
non consummable? In what ways can a high fre- 
quency of practice be provided at low cost? 
Here are the options: 

-i.) Exercises with answers written in the text 
work text. 

* 

Also, see relevant setions below in Stage Four Comments, 
pages 74ff . 
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2.) 



Exercises in a workbook that accompanies 
the text. \ 



3. ) Exercises in text set up so that students 

can easily do them on notepaper/or students 
could write answers in the text. 

4. ) Laminated cards that can be marked then 

erased. 

5. ) Xercx or ditto master^ of exercises that 

teachers can copy as needed. 

2. Should the reteaching come mainly from sources other 
""than student materials? 

a. teacher's presentations lectur^s/deinonstra- 



3. How will mastery be demonstrated? 

a. pencil on paper tests? 

b. actual doing of tasks with teachers observing 
and using check list to score how each signifi- 
cant part of the task was performed. 

c. simulation of tasks with scoring done on observer 
check list. 

Obviously the closer mastery tests are to real-life situ- 
ations, the more certain you can be that the learner will be 
able to apply it. But in situations where learning is restricted 
to the classroom, you will have to develop tests that can be used 
in the classroom yet simulate as closely as possible real-world 
tasks. 



tion of how to do tasks. 



b. 



group interaction discussion, miming, role 
playing, simulation games, actual doing of tasks. 
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Make sure you explore these options with knowledgeable 
outside consultants, or experts in your organization if you 
have any. There will be room to experiment with innova*'jns 
in pilot and field testing. But this could be wasteful if 
an expert consultant could quickly and easily show that inno- 
vations are impracticle. 

Evaluating . the model tests, reteaching lessons, and 
teaching plant . 

The Mastery Learning Panel that evaluated these should be 
expanded with experts in test development and in instructional 
design and development. 

This evaluation should yield an expert critique of the 
model lessons produced. It should also provide judgements of 
features of exemplary bits you have collected. Ask the panel 
"what do you think of this feature?" questions about the 
exemplary materials. 

Here are questions the experts should answer: 

1. What are the good features of the model (or exemplary) 
materials that should be retained? 

2. What features should be eliminated? Why? 

3. What features should be modified? In what ways? Why? 

4. Will this be effective with the intended audience? 

5. Does the material accurately present the subject? 

6. Does the material provide enough instruction and practice 
so users car be expected to achieve the model lesson's 
learning objectives? 

7. Do the tests adequately measure achievement of the learn- 
ing objectives? 
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Set tentative pilot test dates and places 

Ouce the model lessons, tests, and teaching plans are 
evaluated and revised, you will be able to predict when 
the materials will be ready for pilot testing. By that time, 
you will also have a sharper focus on the kinds of students 
for whom the materials will be useful. 

Contact appropriate representatives of those who teach 
or serve this target student population. Through these repre- 
sentatives, contact teachers from whom you can select the 
teachers who will participate in the pilot test. 



Characteristics of pilot test sites: 

1. At least one site per each audience you intend. to 
reach, e.g. one ABE class, one EMH class, one high 
school remedial class. 

2. A range of teacher ability . Highly skilled teachers 
will do many things to compensate for flaws in your 
materials. Note these compensations and incorporate 
them in your materials. Very poor teachers will show 
you how the materials can be misused. This will force < 
you to think creatively on ways to "mistake-proof" your 
materials. 

3. Classes that can devote ample time to using your / 
materials so that the pilot test is not drawn out 

, over a long period. See if you can get teachers to 
double-up -- perhaps allowing up to two hours a day 
on your materials so that you can complete the pilot test 
quickly. But don't push for more than two hours (if 
the materials would normally be used in one hour 
periods). Temper your desire for ^uick results by 
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the desire to see the materials used as they would 
normally be used. 

Classes An which yo u can observe the materials being 
used. The teacher must be willing to accept this. 
Try to find a teacher who would not be threatened or 
who would not put on a show for your sake. The stu- 
d .its, too, must be willing to be observed. In the 
pilot- test, you will want to have observers at each 
site on every occasion the materials are used. There- 
fore the times and places for using the materials must 
be arranged so observers can always be there. 



Why such steady surveillance? You want to 
see how each page of the material is used 
or misused in different kinds of settings. 
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STAGE FOUR: FIRST DRAFTS AND PILOT-TEST PREPARATIONS 
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Assumptions: The Sequenced Test Blueprint is complete and 
has been approved as have also the model 
dtfmain ^specif ications and test items. The 
Model Main and/or Alternate (reteaching) 
lesson(s) are complete and have been approved. 

Project Administration 
The Project Director: 

1. Monitors the test and materials developers and ensures that 
the instructional and testing components of the system are 
congruent. 

2. Arranges for submissions of preliminary tests and materials 
to the Mastery Learning Panel for review and either feedback 
or approval. 

3. Arranges, with the materials developer, for the production 
and printing of the first draft. 

4. Assumes responsibility for preparing the pilot- test sites. 
Field Work 

1. Project Director coordinates the actual kit development 
schedule with the curriculum schedule planning being done by 
the pilot-test teachers. (The dates in the pilot-test sched- 
ule for initiating the kit f s use in each classroom should be 
staggered to allow for ample observation and discussion by 
the Project f s observation team.) 

2. Project Director occasionally conducts, or arranges for, 
interviews with pilot-test teachers to ascertain details of 
how materials and tests should be physically and operationally 
designed to maximize useability in the classroom. 
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3. Project Director assumes responsibility for the designing 
o£ the pilot-test evaluation plan (possibly with the aid of 
an outside expert evaluation consultant). 

4. If there is to be a field test, the Project Director assumes 
responsibility for the development of an outline of the field 
test evaluation plan identifying the types of students and -- 
classrooms to be included in the sampling design. Possible 
sites are selected ^(randomly, if possible) and their teachers 
are approached and invited to participate in the field- test. 
Extensive help from expert professional evaluators is par- 
ticularly essential at this and later stages in the field- 



Test Development 

l u Test developer prepares preliminary items for all knowledge 
(cognitive) tests f and domain specifications for all per- 
formance tests. 

2. Project team reviews these products and decides on modifica- 
tions; test developer modifies accordingly, completes first 
drafts of all tests and their answer keys. 

3. Collaborates with materials developer in writing mastery 
testing sections of the first draft of the Teacher's Guide 
for the kit. 

Materials Development 

1. Materials developer prepares first drafts of the Main and/or 
Alternate (reteaching) texts, including artwork, formatting, 
sketches of forms, etc. ^; 

2. Project team reviews, decides on modifications, and approves 
materials developer modifies accordingly and monitors pro- 
duction and printing of the first draft. 

1 02 
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3. Takes primary responsibility for writing the Teacher's 
Guide; collaborates with test developer in writing the 
mastery testing sections. 

4. Prepares ancillary materials ( teaching aids such as over- 
head projector transparancies , audio/video-cassettes, etc.; 
student aids such as folders to hold all the elements of the 
kit, student Progress Sheets, answer keys for student self- 
correct ion , forms ) . 

WRITING FIRST DRAFT OF TESTS AND TEACHING &ATERIALS 

Congruence between tests and materials 

Since both tests and materials are deliberately designed 
to test or teach the same objectives one might assume that they 
will automatically be congruent. Not necessarily so. 

The materials developers should let the domain specifica- 
tions guide their writing of the teaching presentations. The 
domain specifications include sample questions. They give a 
synopsis of what the test will cover and under what conditions 
the test will begiven. Teaching presentations should be de- 
signed to prepare students for the tests envisioned in the do- 
main specifications. 

But, even with the help of domain specifications, materials 
and tests can still diverge in these ways: 

1. The two developers might give different emphasis to 
different parts of the topic. 

2. The materials developer might choose a format for 
practice items that differs from the formats chosen 
by the test developer for test items. For example, all 
test items may have only one right response in multiple 
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choice questions while practice multiple choice have 
only one wrong response. How important is congruence 
of format? Very! Some students may miss items, not 
because they don't know the subject, but because they 
are thrown by the change in format. If you want 
students to be adaptable to all formats, then teach 
all the formats explicitly, directly, and deliberately. 

Developing reteaching materials for Enhanced Mastery Learning 

In enhanced mastery learning, when a student hasn't mastered 
a unit, he or she goes to a reteaching text. The reteaching text 
presents the same content as the main text, but from a different 
iwrspective. This perspective tends to be more concrete than that 
of the main text. 

In planning reteaching materials, here are some consider- 
ations: 

1. In general, choose a content area that is very interesting 
to your audience. One way to do this is to ensure that the 
content has high utility for them. If you choose an obscure 
topic • Dutch history or the like - you'll need to make the 
presentation or treatment of it very interesting if the 
materials are to succeed. 

If you're not writing the main text, try to chddSe a good 
existing one. If you have to settle for something that's 
rather mediocre, you may want to supplement it with other 
dittoed information or explanations. 

2. The information in the reteaching materials must contain 
the same core information, unit by unit, as the main text. 
To put this another way, the main and reteaching materials 
must each separat / contain all the information covered by 
the mastery tests. 



104 



In general, a team of writers can do a better job than 
one writer* Different people have different skills, and 
a host of skills are required to create good instructional 
materials. Here are some needed skills: teaching of 
literacy skills, communications skills, graphics skills, 
testing skills, sequencing skills... well, it goes on and 
on. A good strategy is to get one main waiter and a few 
editor/writers . 

From the beginning, make it clear to all writers that 
artistry is appreciated, but that artistry mus£ take a back 
seat to function. If a clever turn of phrase or a brilliant 
thought doesn't teach, then it will need*to ^either dis- 
carded or revised. Writers will need to get their egos used 
to that. By the way, allow ample time for revisions. 

Reteaching materials can range in amount of detail from a 
completely new parallel text to a few extra practice exer- 
cises. Two central issues you must face are (1) how much 
reteaching will actually be required, and (2) how much can 
be budgeted for development and printing of the materials? 

i 

If you don f t plan to publish commercially, you may want to 
integrate some existing resource materials (pamphlets, text- 
books, public documents, etc.) into your reteaching materials. 

Don f t spend a lot of money printing your pilot test materials. 
You can almost be sure that whole sections will eventually 
need to be rewritten. < You will learn a lot about your ma- 
terials from the pilot test. 

How teachers will manage the materials is an important early 
consideration. If you plan mastery learning workshops, the 
materials will probably be used as you f d like them to be used. 
If workshops aren't possible, you'll need to figure out ways 
the materials themselves can guide teachers and students into 
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mastery learning. Related to this: you'll need r to decide 
what goes into the teacher's guide and what goes into the 
materials themselves. 

9. Somewhere along the line, you'll need to decide if your re- 
* teaching materials will be consumab 1 e or not, (Material^ 
are jcbnsumable if students are allowed to write in them so 
they cannot be reused by future students';) We chose a middle 
position. Our teacher's guide suggests that teachers have 
students write answers in notebooks -"-unless, of course, $£e 
teachers have a budget to buy new materials for each new 
class. We decided to print the materials so that they are 
reasonably "durable. If we had decided to make them consum- 
able, we'd have opted for printing them on much cheaper 
paper. c 

Writing with the instructional objective in mind 

Hayp^the instructional objective in mind as you begin, writ- 
ing a unit. The objective is what you want the student to be 
able to do at the eiid of the unit. The purpose of the unit's 
content is to help the user achieve this performance goal . So 
ask yourself, "What would it take for the user to learn this?" 

Break the unit down into steps. Each new step should build \ 
on the previous one. Tiy to match difficulty level to the abiliV 
ties (as you perceive them) of the users. As noted earlier, you| 
should try to get to know some typical users ^rior to writing. 
Later on, you'll want to make adjustments in the le^el of diffi- 
culty using data from pilot and field tests. 

As you think through each step in a .unit, consider what in- 
tellectual skills or information you are assuming the user must 
already have. If you realize the user jjrobably doesn't have 
certain skills or information, break the step down into smaller 
steps and teach the prerequisite skills or information first. 
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If you suspecr that a part of a unit you've written pre- 
sents too much material all at once then assume tljat it does. 
It's much better td srr on the side of presenting material too 
thoroughly, than to present material not thoroughly enou'gh. 
Graphics and humor can be a help here. They make what may be a 
slow lesson rabre palatable. ^ 



Appropriate writing * 

J * . 

Most writers who aren't .very familiar with below-grade * level 

readers tend to make materials overly difficult to read. Here* 

are a few good rules of thumb: 

1. Words' should tend to be short and in common use. 

2. J^ite most sentences with ten or fewer words. 

3. Language, when possible, should be concrete. Use 
examples often. 

4. Keep paragraphs short, and separate them with a line 
of space. 

5. Try not to overload pages with text. 

Unfortunately, writers for children also follow these rules. 
So, write in short sentences and simple language. But! DO 
NOT PATRONIZE! Be careful to write as one adult to another. 

Writing with these constraints rarely comes easily. Writers 
should practice and discuss their efforts with the editor. 
Chances are that most of what gets written initially wil 1 need 
extensive editorial revision. As with any other writing, the 
key to success here is to always keep your reader in mind. 



Motivating the reader 

Many low level adult readers have grown up with a. strong 
dislike for school learning. Don't write as if you e.^ect these 
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readers to hang on your every word. Make a concerted effort 
to make your materials interesting and involving. Students 
won T t learn unless they want to and one important function 
of your materials must be to cause students to want to learn. 

Here are some ways to make materials motivating: 

1. From early on, enable learners to appreciate the immediate 
value of this particular instruction to them by giving con- 
crete examples of ways the skills being learned will help 
the student in his or her daily life. 

2. Also, try to orient learners toward the future. Many of 
these students live from day to day. Write about long- 
range goals, consequences, dangers, rewards, and about what 
the skills being learned may eventually lead to. , , 

o 

3. Choose, content and examples which will captute student 
interest. Relevance should be an important concern. Stu- 
dents get involved in their reading when the materials tell 
them something they, personally, really want to know. Once 
again, to write well, you must have empathy with your 
reader. 

4. "Facination" is an exotic word, but it's a good one to keep 
in the back of your mind. Try to make your materials faci- 
nating. Spice them up with the unexpected (but not with 

the obscure and incomprehensible). Do include, occasionally, 
curious irrelevancies . Pose thought-provoking questions: 
"What if...?" Maybe include a story here and there. And 
so on. 9 

5. Use plenty of examples. Remember that these learners often 
lack relevant experience in the topic they are studying. 
Examples help bridge this experience gap. Examples also 
make materials more interesting (ask any journalist). 
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6. Use humor, if it's at all appropriate. Humor tells 
readers that everyone has foibles, not just them. And 
it makes learning more natural... less "academic". And, 
of course, most people enjoy humor for its own sake. 

7. Suggest various motivational group activities: charts to 
make, places to visit, learning games > etc. These can be 
either in the materials themselves, or in the teacher f s 
guide. 

8. Do what you can to help users bridge the gap between 
mastering a skill in the classroom and mastering it in 
real life situations. For example, you might raak^ sug- 
gestions for role-playing, encourage users to try out 
their new skills in specif fc non-school contexts, and 
so on. 

Graphic Design 

c 

The importance of graphics depends on who the intended 
learners will be. If the intended learners are motivated and 
read fairly well, then graphic design can be a secondary con- 
sideration. But if they will probably have learning problems, 
visual design will be important. 

Here are a few design ideas: 

1. Each set of materials will need a consistent format. For 
an analogy, think of the format of any magazine. Page 
after page, issue after issue, there is a consistency: 
department titles, size of photos, style of type, etc. 

But within the format you choose, create variety. 
Notice that there are two dangers here. First, if you don 
have variety, your format may easily become boring. And 
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second, if you let variety overwhelm the underlying format, 
the materials become chaotic and unpleasant. To create a 
successful design you'll need to experiment: As you do, 
tr r to get as much feedback as possible. 

2. Present information in various ways. Use what works best 
for each particular item. Some possibilities: regular text, 
blocked text within a border, charts, graphs, diagrams, 
lists, bold headlines, illustrations, cartoons, photos. 
Always keep in mind that you should present material in a 
way which facilitates memory. 

3. You may want to include non-print media (filmstrips, overhead 
projecturals, records, etc.) with your instructional materials. 
Utility and cost are the main considerations here. 

4. Use graphics for pacing. Pace the students' learning so 
that after new information has been presented, various per- 
spectives on that information are offered. Illustrations 
are useful here. Also, think in terms of visual pace and 
balance. For example, you may want to follow a very crowded 
page with a page containing much white space. And you ma/ 
want to place a cartoon after a fairly long block of text. 

5. Find an artist you can really get along with. Tell the 
artist what you want, but don't be too rigid unless draw- 
ings need to be very technical. Given some leeway, most 
artists will give you some pleasant surprises. Be generous 
in paying him or her. Good graphics can do a lot to im- 
prove texts. 

Format of the reteaching materials 

Rather than try to generalize about possibilities for for- 
mats, we prefer to discuss the actual formats we chose for our 



own Mastering Checking kit. We can fairly well vouch for 
these since we've evaluated them in classrooms, and feel they 
have been reasonably successful. 



Here are some things about the Checking Illustrated format : 

1. Although both can act as main texts, Checking Illustrated 
is design to be slightly easier than Understanding Check - 
ing . We found that the combination of Understanding Check - 
ing as the main text and Checking Illustrated as the re- 
teaching text worked well with middle to upper ABE students 
and with high school students two years below grade level. 

2. Checking Illustrated works well as the main text for upper 
EMH and low ABE students. Checking P ractice is the re- 
teaching text for Checking Illustrated and consists of 
practices, practice tests, and activities. Some of these 
are slightly easier than those found in Checking Illus - 
trated . 

3. Checking Illustrated explains checking in smaller steps 
than Understanding Checking . To do this, it has to con- 
centrate only on teaching basic skills and central concepts. 
Other topics covered in Understanding Checking had to be 
left out to keep the length of Checking Illustrated 
manageable . 

4. Units average about eight pages per unit. Each unit con- 
tains anywhere from one to six or more subsections. 

5. A mastery test appears at the end of each unit. These 
M checkups ff can be used as practice tests or as official 
mastery tests by the teacher. Many teachers may prefer to 
use the "official" mastery tests in the Teacher's Guide 
because answers to the checkups, and to embedded practice 
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questions, are in a separate set of answer keys whereas 
the answer keys for the teacher's versions of the tests 
are in the Teacher f s Guide , 

6. Each unit subsection (generally from one to three pages 
long) ends with a short practice exercise • The format of 
these practices usually parallels the format of the check- 
ups and mastery tests. As often as not, these practices 
are skill oriented. For example, if a subsection teaches 
how to fill out a check, then in the practice, the student 
fills out checks. 

7. The materials include an occasional review unit, followed 
by a review test. 

8. They include a special introductory unit. This unit ex- 
plains mastery learning and explains to the reader how to 
use the materials. 

9. The early lessons move ahead rather slowly, and the tests 
are easy. The purpose is to assure early success. 

10. We placed a heading at the top of almost every page. This 
serves to introduce a page's topic. 

11. We should assume that some teathers won't read the teach- 
er's guide (our field tests bore this out). That being 
the case, it is important to try to make the materials as 
easy to use as possible. Avoid setting up situations where 
students can't know what to do next without instructions 
from the teacher. 

One thing we did was to place a stop sign at the end of 
each unit. This is followed by instructions to the reader 
which guide him or her to the appropriate checkup. 
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12, The material uses various visual devices: diagrams, arrowr 
cartoons, etc. Besides being used for "comic relief 11 , car- 
toons are sometimes used to impart new information. In 
Checking Illustrated , cartoons are sometimes used to illus 
trate process, almost in the manner of a filmstrip. Each 
step of a process is illustrated and captioned, 

13, In Checking Illustrated , few blocks of text contain more 
than four lines. Sentences and words are short. 

Checkups 

"Checkups" are one or two page tests which come at the 
end of each unit. In our materials, additional mastery tests, 
which are similar to the checkups, are in the teacher's guide. 
Answers to checkups are on the checkups 1 reverse side. 

The purpose of these checkups is to give both teacher 
and student frequent feedback on the student's progress. A 
passing score allows the student to go on to the next unit, A 
non-passing score indicates that the student should study re- 
teaching material on the same unit. 

A progress sheet (in the teacher's guide, for xeroxing) 
helps the student keep track of his or her progress. It also 
gives page numbers for both the next unit and for reteaching 
on the same unit. 

Here are a few general rules for writing checkups: 

1. Checkups should test for performances outlined in the 
performance objectives, 

2. In writing a checkup, think first of the actual behavior 
the unit is teaching. As much as possible, the checkup 
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should test for that behavior. This ideal, of course, 
needs to be balanced against the need to not tie up the 
teacher too much of the time. As much as possible, stu- 
dents should be able to learn from the materials themselves. 
This includes being able to correct and learn from their 
own mistakes on exercises. 

3. One pre" .em with self- instructional materials is that they 
can f t respond to students 1 spontaneous comments and ques- 
tions. Therefore, the writer of these materials should 
leave subtle distinctions and ambiguity to teacher-student 
discussions. Try hard not to write text or test items 
where a student could legitimately respond, "No. Whoever 
wrote this is wrong!" Especially steer clear of true and 
false tests. In life, things are rarely always true or 
always false. 

4. Try to write checkups and practice exercises at just the 
right level for users. Know your readers! Checkups which 
are too easy can be insulting. Checkups which are too dif- 
ficult can frustrate readers and destroy their motivation. 
A good test is challenging, but still do-able. 

One of the main purposes of a field test is to 8 ath ^ 
data which will enable you to adjust each test to the right 
level of difficulty. By the way, we found that when tests 
were just challenging enough, students actually looked for- 
ward to them and enjoyed taking them. 

Embedded practice tests 

Embedded practice tests are the little quizzes which 
come at the end of each subsection. You may want to scatter 
additional questions within the actual text of the subsections. 

We cannot emphasize too much how important these little 
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tests are. They are crucial for reteaching because they keep 
the student active . 

One of the reasons students rebel is that they hate 
being passive. The quizzes provide opportunity for action and 
interaction analogous to learning at a computer terminal. 

It is very important that students take these quizzes 
frequently . Without them, poorly motivated students tend to 
either get lost or distracted. Also, since the quizzes allow 
most students to succeed most of the time, passing quizzes gives 
them frequent rewards. Finally, frequent embedded practice 
tests allow teachers to find and remedy student confusions al- 
most at the moiaent they occur. 

We found that many students read without comprehending. 
The quizzes get students into the habit of trying to remember 
and understand what they've just read. The gist of many of the 
questions you ask should be, basically, "What did that last 
passage say?" With these readers, one of the goals of your in- 
struction should be to teach them how to study, think, and 
learn. This is true no matter what the topic of your material 
is. 

Instructional methods we used in our mastery learning materials 

We used two models of instructional design developed by 
David Merrill of the University of Sourhern California and 
Charles Reigeluth of Syracuse University. They have developed 
what are known as the component display theory , and the 
elaboration theory . 

The component-display theory* is a prescriptive theory 
that was developed to integrate existing knowledge about micro 

a 

See Appendix to Stage Four. 
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design considerations (i.e., considerations for teaching a 
single idea). It is comprised of (1) six models of instruc- 
tion, each of which can be used in varying degrees of richness, 
and (2) a unique system -for prescribing those models. The de- 
gree of richness of each model is then prescribed on the basis 
of the difficulty of the objective in relation to the ability 
level of the students. 

Each of the six models of instruction integrates knowl- 
edge about how to optimize instruction for one of six kinds of 
objectives for any given idea: (1) remember an instance ver- 
batim, (2) remember an instance paraphrased, (3) remember a 
generality verbatim, (4) remember a generality paraphrased, 
(5) apply a generality to "new" instances, and (6) discover a 
"new" generality. Each of these six kinds of objectives re- 
quires different instructional strategies to optimize learning. 

For the most common kind of objective applying a gen- 
erality to "new" instances -- this theory calls for three 
teaching strategies: (1) present the generality through the 
statement of a principle or the definition of a concept, 
(2) present examples of the application of that generality to 
specific instances, such as demonstrations of the principle or 
examples of the concept, and (3) have students practice apply- 
ing that generality to new instances, such as solving a new 
problem or classifying a new example of the concept. The prac- 
tice should always be followed with feedback as to whether the 
student's answer was right or wrong and why. The examples and 
practice items should be different from each other in as many 
ways as the student is likely to encounter in the real world; 
and they should be arranged in a progression of difficulty 
from easy to difficult (which may include variation in response 
mode as well as manipulation of variable attributes). Also the 
generality, examples, practice, and feedback should all be 
clearly separated and labeled , as opposed to being in a con- 
tinuous prose passage, in order to facilitate learner control . 
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The second method we used (©laboratory theory) works 
like a zoom-lens camera, Jt gives both a wide-angle panoramic 
view of a subject, zooms in on one particular part of that sub- 
ject to show detail , and next zooms back to give the panoramic 
view and show where the part you just studied fits into the 
whole. The "lens" then zooms in on another part to study de- 
tails, and again pulls back to show where the second part fits 
into the whole or how the second part relates to the first part. 
And so on. 

The example below combines the component display and 
elaboration theories. It starts out with the elaboration 
theory 1 s ff epitomy ,f of a major concept to be taught. We do not 
want to overwhelm learners with detail in the panoramic view 
yet, so we pull back far enough so that one can only see the 
outline, the one main feature trading is giving something to 
get something . This main concept is elaborated by visuals and 
examples. The examples add a bit more detail, so we're gradu- 
ally moving in closer, but still looking at the whole panorama. 

On the next page, we show the whole panorama, but zoom 
in on> one part of the scene trading includes exchanging 
goods for goods . On each subsequent page, we zoom in on another 
p3rt of the whole, thus adding more and more detail to the whole 
In this way, the learner discovers the whole of the subject 
little by little by seeing the whole in relation to its parts 
and seeing the parts in relation to each other. 

The component-display theory (see Appendix) is also used 
in this example. We start out with the presentation of a gen- 
erality (signaled in the lesson with the letter f i f ) which 
states a principle or definition of an idea or concept . The 
letter f e f signals the presentation of an example . Finally, 
the letter f p f signals the practice which students do to apply 
the generality to new instances . (The number in the arrow tells 
learners where to find practice answers.) 
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Trtding is giving something to get something. You give something that 
you have to someone dse. That person gives something he has to you. 



TRADING 



I 



something for something 




money for services 



IP 



What is trading? Mark the correct answer. . 

a. It is seeing the good in others* services. 

b. It. is giving something to get something. 

c. It is giving good service to people. 
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goods 



for 



goods 



TRADING 



I 



something for something 



services 



for 



MONEY 



services 



You've seen that people can trade with goods, services, and money. 

What are goods? Goods are things. You can buy them, own them, and 
sell them. (Buying and setting are what you do when you trade.) 



<5 



These are all goods: 

food records 

clothes jewelry 

stoves soap 

dishes perfume 

toys sofas 

books pttifitw^p 



cars oil 
vacuum sweeper lumber 
bikes steel 
typewriters ships 
desks weapons 
tennis rackets airplanes 




ip i m 



M&rk (**) any items that are NOT goods in these lists. 



4. 



a. cakes 

b. baker 
. c. bread 

. a. diapers 
, b. strollers 

c. babysitting 



2. 



. a. sweaters 
. b. coats 
. c. sales clerk 

. a. football 
b. coaching 
, c. helmet 



3. a. books 

b. writer 

c. newspaper 

$ a. cotton 

b. clouds 

c. woolly sheep 
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The idea is to start with a major concept or procedure 
or principle and then to elaborate upon it until the learner 
fully understands the concept or principle or knows how to do 
the procedure. As you elaborate you don't go off on a tangent 
because you're constantly coming back to the major idea to show 
the relation between the parts and the whole. 

Some students need more elaboration than others. In- 
struction designed according to the elaboration theory model 
provides extra materials and strategies which enrich the elabor 
at ion. If students can master €he objectives with a lean 
approach, fine. Don't bother them with further elaboration. 
But if the lean approach doesn't succeed, you provide further 
elaboration different ways of stating or illustrating the 
main idea, more examples contrasted with nonexamrles, different 
ways of practicing the main idea. The goal is to find some way 
of elaborating the subject so that the student grasps it. Our 
kit provides at least t three ways to teach each learning objec- 
tive. No doubt it doesn't provide precisely the kind of 
elaboration best suited for every learner. But it will come 
closer to matching individual students' styles of learning 
than materials that depend oft only one approach to the subject. 

The component display theory's par titans (i, e, and p) 
separate the instruction into easily distinguishable displays 
of components which aid the learner's control of the learning 
process. If the student learns best by concentrating on gen- 
eralities, he can easily find that component. If he more 
quickly grasps a subject by studying the examples , we made it 
easy for him to find those examples. Perhaps the learner needs 
to interact with the subject through prattice before he can 
master it. Again, he can easily find the component he'll want 
to spend most time with. The labeling helps the learner to 
jump back and forth between practice and generality and ex- 
amples, finding the component wanted at any given stage *m the 
learning experience. 
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By labeling anil distinguishing these thrse major com- 
ponents of learning, we help unveil for the student the 
mystery of learning. Learning is a'matter of understanding 
generalities by looking at examples and nonexaraples of the 
generality. Then you practice applying what you've studied 
to see if you do understand the generality. If assessment of 
your practice shows you don't understand, you know wha^ to 

-- 1) re-study the generality or have it presented differ- 
ently, and/or 2) re- study the examples or have new examples 
and nonexamples given that reveal more completely what the 
generality is and is not, and/or 3) practice applying what 
you've studied in new situations to see if you understand. 

By explaining this process of learning and showing the . 
components in the process to the student, we not only encour- 
age but actually teach the development of independent *study 
skills. If the student is not mastering, he or she knows what 
to do to gain mastery. 

Experiment with writing and design in" the pilot test version 

Be sure that you don't try to perfect the materials in 
the pilot test version. You should have many "which is better?' 
questions. Try two or more seemingly good ways of presenting 
an item, direction, illustration, example or whatever. Let 
pilot test observers, not your personal biases , tell you which 
way worked best. ABOVE ALL don't let your ego get attached 
to what you have written. Stay loose and open. 

If you can't produce and test a variety of ways of pre- 
sentation, pick what you think would be the better way and 
watch during the pilot test to see if it works. If it doesn't 
work, try one of the other options later in the field test or 
in a second pilot test. 

This experimental approach goes against the normal 
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tendency in book publishing enterprises where a group of 
authors and editors sweat for long hours over all the "which 
i£ better" questions, trying to second guess the eventual 
users, in developing pilot test materials you are freed from 
these agonizing sessions. You quickly produce what you think 
will work best or a number of good possibilities and lot the 
users themselves tell you whether you had a good idea or not . 
In the end, the facts are better guides than your very best 
guesses. 

SUGGESTIONS FOR WRITING TEACHING OR RETEACHING MATERIALS 

Provide the message needed in a way that will be heeded 

Deal with issues that are vital to adults and young 
adults. The tone should be adult-to-adult. Stereotypes,, 
regionalism, and condescending tone should be avoided. Though 
the subjects of the books may be serious, handle them with a 
liberal amount of humor. The materials should recognize the 
intelligence, wit, and richness of experience of the advlt 
learner. As developers of materials, we want to provide an 
experience for our readers that is fun, captivating, and infor- 
mative. Above all, the materials should help the reader gain 
competence in whatever subject is covered. The following 
illustration shows how a dry subject can be lightened with a 
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Hook the reader at the start 

Try to start each* book and each chapter with a solid 
attention grabber something to appeal to the reader* s in- 
terests and needs that will make the reader want to con- 
tinue. 

The beginnings of all chapters, topics, and subtopics 
should feature concrete, tangible, known things rather than 
unknown or abstract concepts. In general, proceed from 

simple to complex, 
concrete to abstract 
familiar to unfamiliar. 



Keep it very simple 

No matter how clever, ho bdbk can hold a reluctant or 
handicapped reader if it's too difficult. The reading level 
of our Mastering Checking materials is below 3.6 f >ade level 
as measured by the Gunning Fog Index. No single page went 
above 3.9. Most pages were "below 3.5. Most sentences contained 
less than 10 words. The structure was simple. We tried to use 
no more than two hard words per page. By hard words, we mean 
any word with more than two syllables, except for-- 

1. three- syllable words made up of a two- syllable word find 
one of the following endings: 
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•s, -es 
-•s, -s* 



-ed 
-ing 



•er 
est 



ly 



2. compound words of hree syllables 

3. proper names 

4. clusters of initials -- A.F.L.-C.I.O. 

5. strings of numerals 
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NOTE: Even in our 
use of one -and 
two- syllable word?, 
we tried to use 
familiar words that 
were easy to sound 
out. 



1. Basic checking concepts 
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ImJc checking eonotptt* 5 
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Stress the essentials 

Keep asking: what is the bare-bones minimum a person 
needs to know to achieve the learning objective for this sec* 
tion and to become competent in this subject? It might help 
to develop a list of the absolutely essential topics. Then do 
a second list of very important topics that you'd like to in- 
clude if you have room. Rank the items in this latter list in 
order of importance. Then if you have room for more than the 
bare essentials, you can add the higher-ranked topics from 
your second list. For example, much could be said about the 
advantages of checks over cash, but look at how we've trimmed 
this topic down to bare-bone essentials: 

Show a lot and tell a little 

Present each topic in as few words as possible. Here^s 
where illustrations come to your aid. Depend on them to flesh 
out your topic. Of course, this means you'll hav^to think 
carefully about how to illustrate what you're saying: would a 
cartoon be best here, or a graph, or a photo, or a line drawing, 
or a form? How can I show the reader what I f m trying to say? 



The following illustrations, taken from a section on 
shopping for the best savings account, indicate the kind of 
variety you can use in illustrating" your message. 
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Favorably reflect your audience 

Illustrations, names, anecdotes, and situations 
should reflect a broad range of ethnic and social categories. 
By broad we mean inclusive of all who are likely to be in 
your audience. 



It Amount of money borrowed 
I borrowed $500 at 18%J 




You should have borrowed 
$1,000. The interest is 
12%. 



It's like getting a discount! 

tag} 



for buyi ng a lot of one thing. 

a- a 



Rut there's no use buying a 
tottfyoudon-t**"^ 
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Keep the reader ai ively involved- -reinforcement 

Reinforcement is crucial for all learners, but especi- 
ally for persons in our audience, who need frequent oppor- 
tunities to prove to themselves that they have grasped what was 
reaa. For this reason mastery learning materials provide for 
continuous opportunity for students to demonstrate their grasp 
of what they've read. 

Exercises should be practical 

Exercises should be practical -- filling out the actual 
forms that students will have to fill out in "the real world," 
getting into discussion that will truly help them understand 
or practice what they've read, doing roleplays or other acti- 
vities that will get them familiar with skills that they will 
be needing . 

j 

The best exercise is one that lets the student use a 
skill or demonstrate understanding of a concept in a real- life 
situation. For example, duri:.g-the course, some learners could 
actually go out and shop for a bank. Roleplaying can be the 
next best thing to actually performing a skill. For example, 
students could be asked to roleplay a job interview in which 
one student is the interviewee and another student or the teacher 
is the interviewer. 

Some exercises will be pencil-on-paper. Try for variety 
in this kind of exercise -- multiple choice, matching, fill-in- 
the-blank, puzzles. In these exercises, we may not see skill 
demonstration, but we can at least test the learner's under- 
standing of concepts. A general rule of thumb might be: If a 
section is attempting to teach a skill, e.g., how to write a 
check or how to open a conversation, then use an exercise that 
asks the learner to demonstrate the ability to use that skill. 
If a section is attempting to teach a concept, use a compre- 
hension exercise. 
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Sunuaary description of mastery learning materials for your 
Audience 

In summary, 

present only the essentials of a subject; 
break the essentials down into small topics; 
illustrate or give written examples of almost 
every topic; 

move from simple to complex, concrete to abstract, 

known to unknown; 
provide exercise for each topic, so students can 

demonstrate their grasp of the topic for moving 

to another (this is in addition to mastery tests 

of the end of each chapter) ; 
get the message across with as few words as possible; 
use liberal amounts of humor; 

reflect your audience's ethnic and social composition; 
emphasize practical learning; and 
make the reading level appropriate. 

A short cou rse in graphics 

i 1 " 

Because illustration and design and words are so fully 
integrated, the author must also be an artist-designer to some 
extent. You need to do a rough layout of each page in order 
to know whether your words, illustrations, and exercises will 
all fit on that page. Here are some suggestions to help you 
lay out each page. 

1. Use headings and subheads . The major divisions in 
your book will be marked by chapter titles. Topics 
under each division should be given a heading. And 
sub-topics should be given subheads. Note the size 
of chapter titles, headings, and subheads in this 
section on finding a good savim s account: 
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9. Choosing a checking account 
and bank 

Different kinds of checking accounts 

Some accounts cost money. 
Some Accounts are free. 
Some accounts pty you. 

You might think the best account is die kind that pays. But you need lots 
of money lor accounts that pay If you don't keep enough money in these 
account!: 

• they don't pay. 

• they make you pay. 

So free checking is best for most people . You don't need lots of money 
in the bank. The bank does not charge for its services. Your account is 
free! 

But what if you can't find free checking? Then look for the cheapest ac- 
. count. Banks will help you find their cheapest account. But you must 
ask for it 



2. Start each page with the beginning of a topic or sub - 
topic and finish each page with the end of a topic or 
subtopic . Ideally, we attempt to teach one topic per 
page, illustrate it, and provide an exercise on the 
topic. Thus, the readers get bit-size portions of 
information and are given a chance to taste (illus- 
tration), chew and digest (exercise) each portion 
before taking their next bite. 

Sometimes we stretch a topic over two or three 
ages, with illustrations and examples along the way. 
The exercise then comes on the page on which we've 
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comp leted c overage of the top ic. If a topic stretches 

over more than one page, then no subtopic should be 
split between two pages* Subtopics end on the page on 
which they began. At the top of the next page, you 
begin with a new subtopic which carries a new subhead. 
Here f s an example of a topic that stretches out over 
\£go pages. Note how subtopics end on the page they 
begin on. Also, note how the exercise comes at the 
conclusion of the topic. 



What are checks and why use checks? 



a 



3> 



StaVrr i* the euaoc waaoa people asc checks People don't waai k*a of 
aril ia the* fames or pocket*. The cash could be siolea. So they take thetr 
aeoacy to a beak. Tie beak paoaecai it. Taa book iaaaaci the money That 
ataaas atat ate beak will teniae* any auacy it I 



Bat how do yoa pay for asaacs if *e beak has yoar nwey? Yom give a 
a ri aa e n»eaiperioayoawaotsppay, The iniii a at ia catted a check, h 
arts yoetf hank to pay that pettaa oat 

take* yoar cheek n> your beak. Tfoar beak pa yi thai penoa da; a monat yoa 
wrote eaa yoar check. 

The check cm be aead oeJy by aim penoa foe whom yoa anise it. k*a 
eaasast the law for anyone etee to ate it. The law aad the beak protect your 
aaoaey. fcifcamsaatly. fcsaaodoajy for thoae yoa aril the beak to pay. 




FiH in the bleaks with correct words from this list: 

1. People ase checks because they are 
ihaa rub. 

2. Tb* bank PT** * mn>M y 

3. People fftaaoaey out of the bank by writing 
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Preparing your manuscript . 

Envision each page as it will be in the final book form 
so that you can describe all the elements that should go on 
each page chapter title, headings, subheads, illustrations, 
exercises. Here's how to do it: 

On one sheet, type out a 1,1 the titles, headings, subheads, 
captions, text, and exercises that will be used on a given 
page in the book. Put these elements approximately where 
they will be on the page. The text can be as wide as 60 
characters. (If you set your typewriter margins at 10 
and 70, for example, your line will be 60 characters J.ong- 
maximum width.) If you want text to go alongside of 
illustrations or to be in harrow columns, type it in nar- 
rower columns. (Minimum line length would be about 15 
characters) , See illustration: 



. . _ „ How ds savings and kivsstmsnts differ? 

This paragraph is sat at 60 

characters per line—maximum » S *** ^ **» ***** 

width for a coluan of type. 



These paragraphs are aet at n»/M»fe. Y«'r*t«f» * iufm r%. y M my mi 

a narrower column width. 7 * ywr *** um *•* **y «*«* to* p*n 



m «M «l wktt ym Imvmtud. 



When you want a narrower 
accordingly. You don t ^ m» • w«« tmiT »• ynt MM y. Oti»r 

have tO be exact. You'll Yn t— pi y»r ««M y quickly Imigtwuu In yo« fM your 



Simply be DrOVidina S fttfuUf m—f quickly. TWy inctiid* 

rough estimate of how the 4 t»Mk. 

text will look on the peg** 

Ym frt m mm svofftt Uun th« Ym My §rt Much Higher 

mbAs dhM iSnl from uv- 
Im> lm ysj km to tn%«t te • 
. If y*j ds**t, >ou «ovld 
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More on Graphics for New Readers 

When the artist designs the book he or she should remember 
that our audience is made up of teenagers and adults. "Cute 11 
illustrations on the Bambi level and pictures of children gener- 
ally have no place in our books. We do not want to insult new 
readers by implying that they are dumb or childish. 

Body type 

The type in a book for new readers is important. The 
reader has supposedly just learned how to read, so the type 
should be kept simple and right-sidfc up. Research has proven 
that serifed type faces are easier to read than a sans-serif 
face. Extended faces are easier to read than condensed .faces . 
Medium width faces are easier than bold, which are easier than 
italic faces. 

Words with all letters capitalized are very hard for new 
readers. The new reader gets all kinds of clues from the 
ascenders and descenders. Even headlines should be upper and 
lower case rather than all caps. 

* The size of type for a new reader should be 11 pt type or 
larger. Eleven pt. type should be leaded on 14. The line length 
should not go beyond about 25 picas. If the line is 'longer, the 
new reader 9 s eye will not be able to travel from the extreme 
right back to the ext^^TeTTTfnrnlrr^the correct line. 

Teacher's guides can be in 10 pt type on 11 or 12 leading. 
Ten poini type should not go beyond 20 or 21 picas wide. Eight 
point type can be used for photo or illustration credits, copy- 
right notices, and not much else. Nobody should have to read 
8 pt body copy. 

Captions (cutlines) or directions should be in a different 
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type face to indicate to the new reader that it is not body 
copy. With press roman 11 pt body copy we generally use 
universe 10 pt bold for captions and directions can be done 
in italic. 

Display type 

Display type is usually large enough (24 pt, for example) 
that the new reader can read a sans-serif face. It is not a 
goad idea to use a really large type (like 72 pt.) for a new 
reader because they may not be able to connect the letters into 
words . 

A new reader does not yet have good comprehension skills. 
Sometimes it is too much work for him or her to decode .pages of 
type. Photos, diagrams, and illustrations can provide clues to 
meaning and a break from the work of decoding. 

-The- layout should not look crowded. The artist can work 
with the editor to get a balanced layout. The body type should 
be separated from a cutline by white space so the reader does 
not continue from the text to the cutline. 

Long thin illustrations that y/ut across the middle of the 
page are confusing. The new reader will not know where to 
start reading next. 

Where do I read next? 
New Readers may not have 
good enough comprehension 
skills to figure it out from 
the meaning of the text 




This is intended just as a guide. None of the rules are abso- 
lute; there are always exceptions . We want the new reader to 
discover the joy of reading and books. We want them to want 
to look at books. So basically as an artist, you should try 
to design adult-looking, interesting but not confusing books. 

Printing costs 

New Readers Presi tries to keep pointing costs down so 
that we can pass the books along to the newly literate at low 
prices. Therefore we are dependent on inexpensive design de- 
vices. Lines are decorative, inexpensive, and provide informa- 
tion to the new reader, for example: you are beginning a new 
chapter. Lines can be used many ways for decoration. Type 
cannot. Extra type or type doing unusual things (like slanting, 
mirror effect, or shadowed type) can confuse him or her. Type 
should be used to convey information, not as a decoration. 

Consistency of design is a big help to a new reader. It 
is good for them to be able to identify chapter heads, ]4age 
numbers, cutlines and other elements by their consistent type 
face, size and placement on the page. 

Photos and illustrations 



There are no special rules for, the quality of photos or* 
illustrations. New readers in America are visually rather 
sophisticated. They have been looking at TV, billboards, 
movies, advertisements, etc. They can't read them, but they 
can understand the pictures. They 1 re used to all the graphi- 
cally wild things artists have done. St) you don't have to 
protect them. 

The cropping of a photo should not be confusing. Try to 
not cut off heads, arms, and legs. Do not cut anyone through 
a joint, crotch, breast or anything editorially important. 
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Lay-out 

, White space should be generously and deliberately used. 
White s.^ce and photos can keep a chapter from being an unin- 
teresting mass of gray type, A new reader is not anxious to. 
plunge Into masses of type. 

Why people use checks 




4 



People can't use stolen checks easily. 




APPENDIX 



Excerpts from a handout in the course ITE 600, Syracuse University, 
Syracuse, NY, 1980; Instructor: Dr. Charles Reigeluth 

MODEL OF INSTRUCTION FOR SLOWER LEARNERS 



Then j?res«nt some Then present some 



EXAMPLES 



PRACTICE ITEMS 



Present a 

GENERALITY 

• Knowledge that can 
be applied -in 
a number of speci- 
fic situations. 



G 

Example: 

An adjective is a 
word used with a 
noun or noun equi- 
valent to describe 
something about 
that noun or equi- 
valent. 



-Specific appli- 
cations of that 
knowledge in 
different situ- 
ations 

* E 



tall building 
strong desire 
long game 
It is great 



-Practice in apply- 
ing that knowledge 
in new Situations. 



p 



Underline the 
adjectives in the 
following sentences: 
Bill is very hungry. 
She likes green cars. 
He has a happy face. 



What is an EXAMPLE? 

[GENERALITY j . 

An EXAMPLE is a single , specific object, event, or idea. It 
is a particular one, and there is no other that is exactly 
the same. 



EXAMPLES 



1. My green pen 

2. Shakespeare f s "Ode to 

a Nightingale 19 

3. 4x V Jx+2 

4. (A movie shows a cater- 
piller changing into a 
butterfly. 

5. S.U.'s last basketball game. 
6 Hitting the home run yesterday. 
7. A pen (*♦. is on my desk) 

8 The Great Depression 



NONEXAMPLES 



1. Pens 

2. Sonet 

\ 

3. Quadratic equation 

4. Ijdetamorphosis 

5. Basketball game' 

6. Hitting a ball 

7. A pen (...is needed--any pen) 

8. An economic depression 
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What is a GEN ..\LITY? 



GENERALITY 



A GENERALITY is a set or class of examples that share some 
common characteristics. \ 



EXAMPLES 



NONEXAKPLES 



1. Subtraction 

2. Essay 

3. Anger 

4* Political convention 

5. Volcanic eruption 



1. * 24 - 3 - 21 

2. The essay I wrote yesterday 

3. The way Johnny behaved yester- 
day when I spilt water on him, 

4. The 1966 Democratic Convention 

5. The most recent eruption of 
Mt. Vesuvius 



More About GENERALITY 



GENERALITY 



4 When we use the term GENERALITY, we are usually referring- to 
a statement about a generality, 



A STATEMENT ABOUT a generality should 
explain it. 



fine , describe, or 



EXAMPLES [ 



.1. A sonnet is. a poem that (a) has fourteen li^es, (b) has a 
fixed verse and rhyme scheme, usually iantbic pentameter, 
and (c) expresses a single theme or idea. 

2. To subtract whole numbers without borrowing: 

(1) go to rightmost 'column, * 

(2) use subtraction facts for whole number* from 0 to 9 
to subtract in that column, * ' 

(3) go to next column, 

(4) repeat steps 2 an$ 3 uitil all columns have been done. 

3. Volcanic eruptions occUr whenever the heat and pressure of 
molten rock beneath the surface of the earth becoml great 
enough to break through to the surface. - 

4. A quadratic equation is an equation in which the second 
power, or square/ is the highest power to which the un- 
known quantity is raised. 

5. The law of supply and demand states that in a "free market 
a rise in the price of a good will cause a decline in 
demanu for that good and a rise in the supply of that 
good, and conversely a decline in the price of a good will 
cause a rise in demand tor that good and a decline in the 
supply of that good. 



9 
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More About EXAMPLE 



GENERALITY 

When we use the term EXAMPLE, we are usually referring to a 
statement about an example. 

For the model presented above, a statement about an example 
should show or explain why it is an example of the gener- 
ality- -rF"sKoinT"sHo2^ of the character- 
istics that make it a member of the set or class. 



EXAMPLES 



1. "Look at the following poem (Shakespeare 1 s "Ode to a 
Nightingale")* Look at the number of lines, the verse 
and rhyme scheme, and the number Qf themes or ideas." 
(Pause) "It is a sonnet because it has 14 lines , it 
has fixed verse and rhyme scheme (iambic pentameter 
and ABBA) , and it expresses a single theme or idea. 

2. To subtract 47. 

-23 ' 

(1) First we go to the right-most column, 

(2) then we use subtraction facts to determine that 
7 - 3 - 4, 

(3) then we go to the next column, 

(4) then we use subtraction facts to determine that 
4-2-2- 

So our answer is: 47 

•23 
IT 

3. Mt. Vesuvius had a volcanic eruption 18 years ago because 
the heat and pressure of the lava below the earth's surface 
there built up to the point where it broke through the 
solid cap on top of the volcano. 

4. 3 x 2 - 2 is a quadratic equation because the unknown, x, 
is raised to the second power and no higher, 3 x2, 

5. The amornt of driving Americans did last summer was lower 
than the expected because the price of gasoline had risen 
remarkably. In other words, the demand for gasoline de- 
clined because the price ^rose. 
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What is a PRACTICE ITEM? 

GENERALITY 

/ 

A PRACTICE ITEM requires the student to apply the gener- 
ality to a "new" example, where the example is not 
described or explained to the student. 

EXAMPLES 

1. "Look at the following poem (Shakespeare 1 s "Shall I 
compare thee to a summer day"). Is it a sonnet? 

2. Do the following problem: 72 

-41 

3. Why has Mt. Washington not had a volcanic eruption? 

4. Which of the following are quadratic equations? 

x 3 -~2x 2 - 1 2y 2 + 7y - 3 

Sx 2 - 10 4x 2 
3x ♦ 7 2x 2 + 24 2 - 6* 

5. Why has the production of houses declined recently? 
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What is PRACTICE FEEDBACK? 



GENERALITY 



PRACTICE FEEDBACK is information given to a student after 
doing a practice item. 

This information must include: 



And — if it was incorrect: 

(2) what the student did wrong, and 

(3) how to do it correctly. 

Feedback should ~ome after every practice item . 

Feedback should come immediately after each practice item, 
if possible. 



1. It is a sonnet because (1) it has 14 lines, (2) the verse 
and rhyme scheme are fixed (iambic pentameter and ABBA), 
and (3) it expresses a single theme or 'idea (her beauty) . 

2: 72 This was calculated by (1) going to the rightmost 
-41 column, (2) using subtraction facts to determine 
3T that 3 (3) going to the next column 7, and 



3. Because the lava is so far from\he surface that it cannot 
build up sufficient heat and pressure to break through to 
the surface. 

4. x^ - 2x 2 - 1 is not a quadratic equation because the unknown 
is raised to a power higher than 2 (x 3 ) . 

5x 2 - 10 is a quadratic equation because the unknown is 
raised to a power of 2 (xZ) and no higher.' 



5. Because the price of houses has declined (a drop in price 
causes a dfop 5n supply or production).' (The drop in price 
was also caused by a drop in deman,d for houses, which was 
caused by high interest rates.) 



(1) whether the student's work was correct or not, 



EXAMPL E | 



(4) using subtraction facts to determine 
that 7 




etc. 
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STAGE FIVE: CARRY OUT THE PILOT TEST 

1. The Project Team collectively: 

a. Projects when the teaching materials and tests will 
be produced and copies printed for participants in 
the pilot test. 

Revises and finalizes the sampling design which: 



b. 



2. 



1. 



JelimifS the major broad types or categories of 
students and classrooms that make up the ultimate 
future student population that are intended to 
use and benefit from your new mastery learning 
system. 

2. Specifies the classes that will serve in the 
pilot test as representatives of each category 
(preferably include three representative or 
typical classrooms in each category) . 

The Project Director notifies the participating teach- 
ers of those classes about the project schedule for 

a. Starting-up the use of the first draft in each 
participating classroom. 

b. Visits to the class by the observation team prior 
to those start-up dates. 

These prior visits will enable the observation team to 

1. Let students get acquainted with the members of 
the observation team and used to this new experi- 
ence of participant observation. 

2. Observe how students normally interact with 
regular printed materials used in those class- 
rooms . 

3. Teach individual students to candidly express * 
their thoughts out loud to individual members 
of the observation team while they 1 re studying 
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and taking tests without developing dependencies 
on the observer for feelp. / 

(Note: the schedule may have to stagger the start-up 
dates at some of the sites if that is the only way for 
each class' first week using the first draft to be com- 
pletely observed by the observation team.) 

The first draft is carefully edited and copies are made 
and packaged in a form that approximates the look and 
feel of the materials as they will Oltiraately be pro- 
duced. 

The Project Team collectively discusses issues that they 
think at this preliminary time they 'should keep in mind 
when they eventually start observing the first draft in 
action in the pilot test classes. (See suggested ex- 
amples of these issues in the Appendix to this Stage.) 
This is the beginning of the evolving "Observation Check- 
list" that will be continually re-evaluated and revised 
over the course of the pilot test. 

Just prior to the start-up dates, participating teachers 
should be briefed on Exactly what*is expected of them s 
and how you want them to help you. ,Mak$ them part of the 
team and invite them to plan on sitting down with the ob- 
servation team a£ter many of the observation sessions to 
discuss what happened, and to brainstorm ways to improve 
the section of your first draft that was used that day. 

Observation team leader (s) delivers, each participating 
teacher his/her class set of, the first draft, Do not 
explain what you are delivering or ; otherwise help the 
teacher in any way beyond what you expect future teach- 
ers in your target population will themselves be able 
to learn about your materials from whatever advertise- 
ments you expect you will be distributing or from other 
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kinds of announcements, presentations, teacher- 
training workshops, etc. 

7. The observation team(s) conduct site visits using 
multiple participant observation methods. They should 
also carry out well prepared and well planned, but 
brief, interviews of each teacher at the end of the 
class {if possible) or soon after while things are 
still fresh. Ask teachers to identify problems and 
suggest solutions. Team members should take copious 
notes to record their observations and should discuss 
among themselves their respective observations and 
interpretations regarding both the issues in the Ob- 
servation Checklist and new issues that have arisen 

in their minds. Observation Checklist issues are 
annotated with the day 9 s findings or are deleted if 
found to be irrelevant; new issues are added if the 
team so decides. 

8. When all sites have completed the course, bring par* 
ticipating teachers together for a wrap-up discussion 
of outstanding issues. Get them to identify the most 
severe problems and to suggest and evaluate alternative 

^solutions f * 

9. The Project Team collectively assesses all findings of 
the pilot test based on their o*rn copious notes, te^rhe 
suggestions, and teacher contributions during the wrap- 
up 1 discussion. The team condenses and refines formula- 
tions of the most important feasible proposals for re- 
vising the materials and tests. 

COMMENTS ABOUT THE PILOT TEST 

The pilot test gives you a chance to find out if your cre- 
ation really works as Well with students and teachers as you 
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have been hoping and predicting. In creating the first draft, 
the developers have tapped their own experiences and talents as 
well as the gobd advice of others. But their beliefs about what 
will help students learn actually amount to good guesses and 
speculations • 

The pilot test injects trial-and-error experimentation into 
your development process. The first draft of the materials and 
_^tests represents your best bet, but it is still a gamble. It's 
'a proposal you are putting on trial. Look at the pilot test as 
a dry-run laboratory experiment for detecting the bugs and prob- 
lems you cannot see when sitting at your desk. Don't look at 
the first draft as the final perfected product of your labors. 
Spend less time at your desk agonizing over fine points in the 
first draft, and spend more time observing students and teachers 
interact with your creation. 

Remember that a pilot test is not classical research. You 
are not trying to grade or rate tlje effectiveness of ypur ma- 
terials in comparison to rival materials or teaching strategies. 
In a pilot test f you assume that your materials are generally OK; 
your goal is to find out how to make them even better. 

If you work at it f you will be able to make your pilot 
test the most significant and rewarding part of your whole pro- 
ject. Here are some suggestions for making this come true based 
on our own experiences in* our pilot test. 

FIRST, remember that you would like to find out what stu- 
dents ind teachers are really thinking when they interact with 
your first draft. But, since you cannot telepathically eavesdrop 
on their private thoughts, ; you should try to gather as much 
"objective" and publicly observable evidence as you can about 
what they are thinking. 

The observation strategy we urge you to adopt is a strong 
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form of multiple participant observation . Forget about 
"Hawthorne Effects" and other impacts your observations have on 
students when you interact with them. You are trying to detect 
differences among the impacts on students caused by differences 
among the elements of your first draft. These differential im- 
pacts will still surface whether or not your presence makes 
students feel abnormally alert and motivated and want to please 
you. 

On the other hand, "participant observation does not mean 
that you should do more than observe . Observe without giving 
students or teachers special help when your materials fail. Do 
not present yourselves as experts on whom students or teachers 
can become dependent. Don't rescue them when they get into 
trouble. (You might help them that time, but in doing so you 
might also be training them how to overcome the same defect when 
it recurs repeatedly later on. The next time they encounter the 
defect they will know how to overcome it, but you might overlook 
it on that and all later occasions. Future users of your ma- 
terials will suffer because you did not know to correct those 
later defects.) 

You can't be kind to pilot test participants without being 
unkind to many more unseen future users. And you will end up 
with inferior materials so, observe , but don't help . 

SECOND , visit pilot test classrooms several times before 
the teachers start teaching your first draft so you can develop 
rapport with the students and enable students and teachers to 
become accustomed to your participation. Get teacher permission 
to approach and interact, with students when they are working on 
their own with other materials. In sum, construct the desired 
relationship with the pilot test classes before your materials 
are introduced. 

As a corollary, remember that you are "teaching" students 
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how to freely converse with you, to think out loud to you , while 
they are interacting with printed materials. Tell them that 
that is what you would like them to try to do, to freely think 
out loud; of course, whenever they do appear to be trying to do 
this, reward them regardless of the content of what they actually 
say. Make positive reinforcement the rule, 

THIRD , ensure that "observation teams" are composed of at 
least three members of your project team. A minimum of three is 
required because different observers will be sensitive to dif- 
ferent aspects of classroom interactions and will relate dif- 
ferently to different kinds of students. Three or more ob- 
servers in each classroom observation session will ensure that 
most of the deficiencies in your first draft *re detected. Also, 
individual biases among three or more observers will tend to 
cancel out each other thereby increasing the "objectivity" of 
your findings. 

FOURTH , pay close attention to how teachers score the mas- 
tery tests at the end of each lesson and then make decisions 
about whether the student has mastered the lesson and should 
move on to the next lesson or has not mastered the lesson and 
should instead relearn the lesson's content by studying a re- 
teaching lesson or doing extra practice. You want to watch for 
ways to make your materials discipline teachers so that they 
neither prematurely advance non-masters nor hold back true 
masters. 

FIFTH , be on the look-out for times when your mastery tests 
say that certain students are masters or non-masters who teach- 
ers "are sure" are not true masters (even though they "passed 
the test") or "are sure" are ^genuine non-masters (even though 
they "did not pass the test") ; the teachers may be right and your 
test may be rendering false judgements. 

Remind teachers (and yourselves) that each prototype test 
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is being put on trial . It is not necessarily better than 
the teacher's own intuitions and judgements. Take special pains 
to ask teachers which tests seem to produce results that are at 
odds with their own assessments of their students. 

Generally make teachers feel that they are part of your 
team and persuade them that they share in, and have a stake in, 
the whole enterprise. Interview them after each observation 
session and convey to them that you are taking their testimony 
very seriously. Only they really know individual students in 
their classes and can spot things your observers overlook or 
misinterpret. 

SIXTH » observe, and ask teachers about afterwards, how 
manageable your materials Are in the context of normal class- 
kpom dynamics. Our own kit was at the cutting edge or unmanage- 
abllity. We. gave students three texts under separate cover to 
choose from for learning the same lesson. Each lesson in each 
of these three texts had its own exercises and its own mastery 
test, ^eachers had to grade both the different exercises and 
the different tests individually for each student. Also, stu- 
dents worked with separate forms (sample blank checks, deposit 
slips, etc.). To make things even more complicated, each 
student moved through the texts at his or her own pace, so stu- 
dents in the class were generally working at different places 
in the course. Finally, each student had his or her own "Stu- 
dent Progress Sheet 11 on which the teacher had to record whether 
the student passed or did not pass each particular mastery test 
and had to indicate which of the three teaching texts the stu- 
-ent should next move to depending on how he or'she did on the 
mastery test. 

This was kind of a "worst case 11 maangeability arrangement for 
the teacher. Our observations in the pilot test led us to radi- 
cally simplify the system and reduce the management problems for 
teachers. You also will want to observe teachers using your 
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materials in your pilot test to learn how you can make your own 
materials more manageable and efficient without compromising 
their instructional integrity and effectiveness. 

SEVENTH , spend time before the onset of the pilot test to 
work up a preliminary checklist of questions and issues for ob- 
servers to keep in mind while they are observing students and 
teachers interact with your first draft. Continually revise, 
refine, and update this observation checklist so that it be- 
comes a diary or journal of the pilot test as weli as a contin- 
ually improving record of your findings. This process will not 
only make members of your observation team more alert to the 
broad variety of potential trouble-spots they might encounter, 
but will also tend to consolidate the individual viewpoints of 
separate observers -into a common project -wide perspective. This 
process will additionally help you separate problems with par - 
ticular words, sentences, and paragraphs in the teaching ma- 
terials, and problems with particular test directions and test 
items, from systematic problems that recur throughout longer 
sections of the first draft and require systematic correction. 

For examples of issues you might want to consider for your 
own observation checklist, see the Appendix to this section. 

EIGHTH , You have hopefully selected classes for participa- 
tion in the pilot tost' which meet the following requirements: 

1. The classes are typical or representative of the classes 01& 
groupings of students that /cu intend your materials to be 
used by in the future. Pilot- test students should resemble 
your target future student audience in ability level, level 
of educational attainment, and socio-cultural and demogra-* 
phic characteristics. 

2. Pilot test classrooms should also resemble your targeted 
future classrooms in the size of the classes, the- number 
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of teachers/aides/paraprofessionals present in the class- 
room, and the physical and scheduling characteristics of 
the host school or institution, ^ 

Divide your targeted students and classrooms into salient 
types or categories. For each type, find three classrooms 
which resemble that type to participate in the pilot test, 

3. Within each type, try to choose three teachers who appear to 
represent a cross-section of ability and teaching style. The 
least effective teachers will provide an acid test of your 
Teacher's Guid<e and the manageability of your first draft. 
The most effective teachers will devise ways to correct or 
compensate for deficiencies in your materials that will give 
you good ideas about how to improve them, 

4. Ensure that you havfe a manageable and not too numerous 
group of pilot test sites. You should try to have an ob- 
servation team observe each site's first week of work with, 
your first draft, so you should staggeT the initiations of 
the course at the different sites. Your project team should 
expect to spend 75* of total project time during this pilot , 
test period sitting in those classrooms and observing your 
materials in action. A good ratio is one observation team 
(with three project team members) for three or four sites. 
After each site has moved beyond its first week with your 
materials, ensure that the team visits it at least once 
during each subsequent week for the duration of the course. 

5. Finally, ensure that your materials are pilot tested during 
a fairly "normal" period in the school year, not at the be- 
ginning or end of the year (semester) or near Christinas. 

NINTH , try to produce your first draft at as high a 
level of production quality as you can afford to make it. Use 
good print quality and good binding. No typoes, handwritten 
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words, crossed-out sentences. Even though it is a "first 
draft," it should look like a finished product to the extent 
that you cah afford to do this. If you don't do ttfis. it will 
be very difficult for you to determine whether students r and 
teachers in the pilot test are having problems or expressing 
confusion or negative reactions because of defects in your 
instructional system , or because of rough or clumsy physical 
defects, in your first draft manuscript . 

Recommendations from the leader \of our own observation team 

Have the observation team discuss what to look for during 
the pilot test. Watch out especially for gross problems in the 
tests--confusing directions, test items that were confusing or 
failed to clearly indicate student's ability, test administra- 
tion problems. 

« 

Observ^ each session when pilot test materials are used. 
Have vour presence explained as people who are developing ^ma- 
terials and want to find out how to improve them. This honest 
explanation might influence how students perform, yet this is 
better than forcing teachers to say you are something that you 
are no{. 

* • 
Rather than remotely observe, be participant observers. 
When you see a student in trouble, go to the student and ask 
if you can help. This gives you direct contact with the stu- 
dents, produces accurate knowledge of what is causing problems , 
*and reveals why a particular paragraph, direction, dr test 
item is problematic. 

Take notes on teacher and student comments about the ma- 
terials. After each session, have members of your observation 
team discuss their obsei^vat ions . Take notes on t.h%se debrief- - 
ing sessions. 
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In the beginning, place no restrictions on K ow the teacher 
uses the material other than that they follow suggestions given 
in the Teacher f s Guide. Note who does and does not read and . 
easily follow the Guide f s suggestions and correctly administers 
the teaching and testing. Try to understand why some get^con- 
fused by your kit, or don f t know how to manage the teaching and 
testing. nr own pilot test was very valuable in that it pre- 
pared us for seeing what had to be done to make the materials 
"mistake-proof" for teachers, and much easier to manage. 

Ask students to keep all materials and tests they use in 
their folders, and to not take these folders home. Let teach- 
ers know, that you will want to go through the folders when t> 
course is completed. 
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APPENDIX 

SUGGESTI ONS OF ISSUES FOR PILOT TEST OBSERVERS 

Do the teaching materials match studeat^ahility levels? 

1, Is the reading level (vocabulary, sentence and para- 
graph complexity) appropriate? (Too high a^ level will 
discourage students, too easy may offend them.) Can 
most problems here be solved by defining more words 
and giving students repeated practice in their mean- 
ings? 

2, Is the range and flexibility sufficient? Do the ma- 
terials as a whole provide enough variety to meet the 
needs of both the fastest learners and the slowest 
learners? 

3, Where and when do students appear to get confused or to 
most frequently need individualized help from the teach- 
er? Which lessons are too compact or sparse and need 

to be broken down into more and easier steps? Which 
lessons are just too hard? Which directions are un- 
clear? Is the material organized on the page in a 
confusing way? Are pictures, illustrations, graphs, 
diagrams, or their captions confusing? 

4, Which elements or lessons seem to be the most and the 
least effective? Can the differences in effectiveness 
between these lessons be associated with differences 
in their teaching method, in how they present the con- 
tent, or only with differences in the actual difficulty 
of their respective contents? 

5, Is the learning sequence right, or should lessons be 
shuffled or reordered to put harder lessons nearer to 
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the end of the course? Are all le r ssons indispen- 
sible, or can one or more be dispensed with? 

6. Are there enough review sections, and are they* placed 
at the right junctures? 

i 

! 

7. Which kinds of students and teachers do the best and 
the worst with the materials? Should you narrow down 
or expand the intended range of types of students for 
whom these materials will be appropriate? 

8. For different types of students, what is the optimal 
proportion of narrative to illustrations? -to exer- 
cises? -to tests? -to page length? -to lesson 
length? 

9. Which elements in the content seem most to need whole- 
class discussion and- teacher demonstration? Which ele- 
ments might be helped by teacher explanations using 
overhead projector transparancies? Which elements 
might be helped through group simulations of real-life 
situations requiring special materials such as fake 
money used in simulating transactions in our bank check- 
ing course? Which elements might be helped through 
uses of other technologies such as\reel- to-reel films, 
film strips, ITV, audio recordings, etc.? 

10. Do exercises and practices adequately prepare students 
for the tests? Do exercise and test items have the same 
structure, use the same vocabulary, cover the same con- 
tent? 

11. Do exercises cover all but only the content that students 
will be tested on? Are answers to all exercises ade- 
quately provided and explained in a near-by narrative? 
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12. Why do Students give incorrect answers to exercise 
items? Are they skipping over key sections in tHe 
narrative? Are explanations in the narrative poorly 
formulated or unclear or too hard? Are answers in the 
narrative too far removed from the exercises (two pages 
back, two lessons back, etc.)? Are the directions or 
the structures of exercises confusing? Finally, are 
the exercise items themselves poorly formulated? 

13. Are the tests adequate? Are they too long? Are they 
threatening? -if so, why? Are students failing be- 
cause tests cover too much? Because directions are 
poor or test items are poorly formulated? Are students 
passing who have not really mastered the material? Be- 
cause it is easy to cheat? Because the test covers 
only one part (the easiest part) of the lesson? Be- 
cause, the way they 1 re formulated, the test items 
answer themselves? 

14. Do students need more aids (e.g., samples of items used 
in real-life situations, learning games, etc.)? Do 
they need more exercises and practices to prepare them 
for the mastery tests? 

15. Are the principal learning materials (the "Main Text") 
just too difficult for a sizeable number of students? 
Do they need a simpler or more motivating "Main Text 11 , 
and accordingly, an even easier and longer set of 
practices to use for relearning the content when they 
don*t pass the mastery test? Can you develop a separ- 
ate mastery learning system for those students? 

16. How many forms, or equivalent but different versions, 
of each lesson's mastery test are needed for retesting 
non-masters or for ensuring that students can not copy 
each other f s answers? 
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17. Do students desire (and do teachers need, and will 
they accept) student answer keys to exercise and/or 
mastery tests so that students can correct and score 
their own exercises and/or tests (and save the teacher 
a lot of time)? 

B. Do the materials sufficiently motivate students? 

1. If you observe students working in other materials 

prior to the introduction of your first draft, do they 
appear to work in yours with comparatively more/ less 
interest, more/less tendency to be distracted, daydream, 
become discipline problems? Check your bef ore-and-af ter 
comparisons with those of the teacher. 



2. What spontaneous comments or other behavioral indi- 
cators of interest do students evince? 

3. Do students appear to like or to not like working on 
their own at their own pace with only occasional indi- 
vidualized help from the teacher? 

4. Ask students which pages, elements, lessons they like 
doing the most. Find out why they prefer these to 
others. 

5. Which tests did they like doing the most or dislike the 
least? Why? 

6. Do students prefer this or that kind of exercise or 
test item? . Because it's easier or why? 

7. Do students like frequent but brief tests, or would they 
prefer fewer but longer tests, or just one big test at 
the end? 
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8, Do they prefer pages witfr less narrative and more 
illustrations (e.g., cartoons), or pages with fewer 
illustrations and more narrative that better explains 
the content? 



9. Do they appear to prefer some topics, contents, subjects 
to others and, if so, in which cases might this be more 
due to the method of presenting or teaching the content 
than to the content itself? (In any case, solicit 
student preferences to identify sections you will need 
to make more exciting, amusing, etc.). 

10. Are students who fail mastery tests glad that they have 
a second chance to relearn and master the material? 
About how many relearning-retesting cycles canstudents 
of different types in different kinds of classrooms go 
through before they feel discquraged or humiliated and 
obviously need to be advanced to the next lesson whether 
they achieve mastery on the present lesson or not? For 
those students who do not pass the first *ime around, 
does ultimately passing at the true master's level 
sufficiently offset the feeling that they 1 re getting 
behind or not as good as others that they face the next 
lesson more with eagerness than with trepedation or 
discouragement? 

11. Does student interest seem to rncfease or wane during 
the course? As time goes ort, do students increasingly 
try harder and more eagerly await mastering the next 
lesson and the next test? (Make sure that earlier tests 
are relatively easier than later tests to give students 
plenty of rewards at the outset and increase self- 
confidence.) 
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C. If individualized learning is a goal, how is it working? 

Does the teacher encourage questions, teach small groups, 
look comfortable (not harried) and feel more at ease as 
time goes on? Do^students constant ly reques t-ex planatio ns 
of the content or what to do next (because your materials 
do not adequately explain themselves)? 

(In our pilot test of our own materials, we found that the 
less skilled the student, the more work that was required 
of the teacher and the more need there was for whole-class 
instruction.) If whole-class instruction is needed fre- 
quently, are there Ways your materials can encourage the 
teacher to .involve most students and prevent some students 
from tuning out? 

Does the teacher use a variety of teaching strategies? If 
not, do things get monotonous? Can you add a greater 
variety to your materials to offset this? < 



D. Are the materials manageable? 

i 

1* If your materials either consist of several separate 
parts or are designed for students to move individually 
along different pachs (sequences of lessons or units) 
through your materials, then: 

a. Can the teacher easily manipulate and orchestrate 
student use of the material^ and keep track of all 
interactions between students and the materials? 

b. Are more directions to students needed, more direc- 
tions to the teacher needed? Or do the materials 
and the system you've devised need to be stream- 
lined and simplified? 

c. Do students frequently ask "What am I supposed to 
do now? ,f 
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d. Does the system begin working smoothly after some 
initial chaos? To make it work smoothly, has the 
teacher had to. devise his/her own strategy for 
using the ma terials which strongly differs from 
your intended strategy and utterly destroys the 
value and validity of the system? 

e. Ask teachers_how they would use your materials if 
your observation team was not in the room. (Some 
teacher suggestions may point to ways to really 
-improve your system. In any case, you must de- 
sign the system, or present it, in a way that is 
truly acceptable to teachers since you will eventu- 
ally not be able to control how they use it. J 

f. Do students, or do teachers, move around the 
classroom more? Which appears to work better? 

g. Do teachers who read your Teacher* s Guide still 
seem confused about how to use your materials? 
Why do those who don't read the Teacher 1 s Guide 
refuse to jtead it? (Would they have rdad it if you 
had made it more eye-catching, attention-grabbing, 
easier to consume quickly?) 

At what class si£e does it become very difficult for 
teachers to properly manage the use of the materials? 
What class size appears to be ideal? Does this square 
with assumptions you have been making about your target 
students and . the size of their classes? Can larger 
classes be effectively divided into small groups or 
pairs of students? Can teacher aides or students them- 
selves assume some of the management responsibility. 

Do teachers or students seem to ignore or overlook some 
part of your materials? Can you make that part more 
conspicuous, prominent, or central? Or is that part 
actually expendable? 
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4. What happens while student mastery tests are being cor- 
rected and scored by the teacher or other grader? Is 
there a sizable time-lag in this feedback loop? Do 
students use this time effectively? Is having students 
correct and score their own tests an alternative solu- 
tion? Could they use the time skimming and preparing 
for the next lesson? Could they be helping other 
students? 

5. Do some teachers manage better ttyan others? Is this 
because of personal differences between these types of 
teachers or the types of classes they teach? Or are 
the more effective teachers using management techniques 
that can be taught and transferred to the less effec- 
tive teachers? .Can you incorporate the teaching of 
those techniques in the Teacher f s Guide? 

6. Do some elements or lessons seem to work better with 
one kind of classroom organization than with another? 
For instance, does abstract content seem to be best 
handled by whole-class lecturing led by the teacher? 
Which elements and lessons are handled well when stu- 
dents work on their own? Which work better in small 
groups? Which work better in role-play simulations? 
Which just have to be handled by aides or teachers 
working individually with the student? 

Finally ^does the teacher, and do the students, seem to 
GRASP the concept and process of Mastery Learning ? 

Do they understand that the tests are for determining 
whether students have mastered each lesson and are ready 
to move on, or for identifying deficiencies in their 
learning to focus on in their relearning of the material? 
Do they grasp that students compete with themselves for 
mastery, not with each other for grades? Do they grasp 
how to use the reteaching materials? Do they emphasize 
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and appreciate the fact that non-masters always get 
another chance v a real chance with no "criminal record" 
held against them, until they ultimately become real 
masters? Do they grasp and prac tice individualiz ed, self- 
paced, independent learning and self-instruction? 

If they do not grasp these things, then your Teacher's 
Guide, and the students' materials as veil, may need to be 
augmented with an introductory lesson that teaches these 
things to teachers and students before they start the 
course. 
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STAGE SIX: REVISE PILOT-TESTED FIRST DRAFT TESTS AND MATERIALS 

1. Based on Stage Five's revision priorities and on avail- £ 
able resources, assign revision tasks and staff and- 
develop a schedule indicating by when revisions oT tests", 
teaching materials, and the instructional management 
system are to be completed. 

2. For all proposed solutions to g^oblems uncovered in the 
pilot test, and especially solutions to instructional 
management problems, present the solution ideas to 
pilot test teachers and to memhers of the Mastery Learn- 
ing Panel and solicit their judgements and preferences. 

3. In. regard to a major proposed solution for which there 
is not availabl supporting evidence about its worth, 
test it by comparing it to one or more alternatives. 
For instance, .suppose the problem is that some pilot 
test teachers did not pick up and read the Teacher's 
Guide, and the proposed solution includes a particular 
way to make the cover of the Guide .nore eye-catching 
and specific ideas to make the first few pages of the 
Guide easier to read and more effective in presenting 
key concepts. Then a tes£ of this proposal might con- 
sist in asking five teachers of target students who, 
were not in the pilot test to compare a mock up of this 
proposed solution to teacher's guides in other pub- 
lished kits or series and also to mock ups of one or 
two alternative covers with ideas for revising the 
first few pages. 

. Before fully implementing major revision proposals that 
have been approved by the whole Project Team, cost them 
out to ensure that you can not only perform them and 
still stay within budget but can also still produce the 
final products at reasonable cost levels. 
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Revisions in our materials based on both our pilot test 
and our field test 

1. A major problem in our pilot test concerned the manage- 
ability of our materials. Some portion? of them, being 

difficult to manage, simply didn't get used by some 
teachers. We made the mistake of boxiiig lessons and 
tests lotfse in a file system. This meant ^ 
always a lot of loose paper around, and that teachers or 
students would have to be continually going to tus box for 
materials. So after the pilot test, we went to a text 
(book) format except for tests. (After the field test, 
we decided to even include tests in books) . 

2. The pilot test showed that some teachers weren't grasping 
"mastery learning" and were using the materials to teach in 
their usual style. The best* solution to their problem 
would 've probably beec some sort of in-service testing for 
teachers on. mastery learning. B t ut, since our goal is a 
published kit that needs no such supports, that wasn't an 
option. Decisions we did make: improve the teacher's guide 
to make it more readable and to emphasize the system more; 
place very clear instructions on what to do at every de- 
cision point in the materials (leaving nothing to the 
imagination); and add preliminary lessons in student ma- 
terials on mastery learning. 

3. The materials contained some features' whose value needed 

to be analyzed. Analysis came from observation, interviews 
and teacher questionnaire responses. For example, we needed 
to know if the illustrations in the student texts were 
appropriate for this audience, and, if so, was the amount 
of illustration appropriate. Revisions were made accord- 
ing to feedback (in the case of illustrations, no signifi- 
cant changes turned out to be warranted) . 
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There'd been a question on the number, placement, and 
appropriateness of the mastery tests in the student texts. 
The best data here came from observation ancf analysis of 
the tests students took. We found that there were a £ew 
too many tests, so the total number was reduced by com- 
bining several of them. Also, a number of students did 
very poorly on two particular tests. We realized that the 
instructional preparations for those tests was inadequate, 
and consequently made extensive 1 revisions in the correspond- 
ing teaching materials. We decided .that only minor re- 
visions were required in those tests themselves. 

Fairly extensive changes in vocabulary were made as a result 
of the pilot test. Basically, we'd aimed too high, and we f d 
assumed students would know many words which they didn't 
know. So some of the harder words were taken out and simpler 
words substituted. A glossary -defining the remaining hard 
terms was added to the student texts for quick reference. ' 

Some students required a lot of practice. Additional tests 
were later added and the Checking Illustrated text was ex- 
panded (from 7C to 96 pages) by adding even more explana- 
tion. Also, based on our pilot test observations, . some • 
students were learning slightly above the level of the 
materials. B Fot that reason, and for reasons of economy, 
the Understanding Checking text was shortened a bit (some 
explanation was removed - the lesson structure was re- 
tained to make all three books have lesson-to- lesson 
correspondence to each other) . 

One lesson in the materials ( ft choosing an account and 
bank 11 ) turned out to be much harder for students than 
we'd expected. After the pilot test, we decided to move 
it from early in the materials and make it the last 
lesson. 
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Setting up the field test was easy. Teacher? were eager 

to get free materials. 

< 

We determined from the pilot tens t that the inateTials were, in- 



appropriate for English-fbr-Speakers-of-Other^Languages 
students. We discussed whether or not we could make them 
appropriate, but droned that idea as unworkable. 

We had one pilot test class with 15 students*. That was too 
many. The later field 'tests were done with smaller classes 
(except for one) . *> ^ 

(By the way, relevant to nothing* That large, class, had 
three aides. Still, the teacher insisted on lecturing. 
She had to maintain -her status* Had they tjYoken up into 
groups, on the surface at least, she'd l^ave>had the same 
status as the aides.) 



By continually reexamining our field notes!, ^we were able to - 
keep track of particular problems with it.ems in the tests and 
materials. These problems were tUen corrected. 
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STAGES SEVEN - TEN: Field Test, Final Revisions, Production , 
and Publication and Marketing of the Materials 

Our own development project included four additional stages 
which will not be described here: 

Stage Seven: Field-Test of Revised Materials and Tests 

Do this very expensive and complex stage only if 
* there is a strong need for formal statistical 
evaluations of your mastery tests. 

Assess the tests to detormine their reliability and 
to perform item analysis on test items to identify 
those items for deletion that do not accurately sort 
students into master and non-mastfer groups. 

Compare types/categories of students to one another 
statistically to identify those types with whom each 
component of the kit is the most and the least effec- 
tive. This enables you to identify which types of 
students may need extra help and support, and to 
identify which components of the kit are best suited 
for different types of students. 

The field-test should be performed with a minimum of 
200-300 students at ten or more sites who actually 
complete their learning in your materials. If your 
target population includes more than one major type 
or category of students, then multiply this sample 
size by the y total number of target categories. 

Stage Eight: Final Revisions Based on Field-Test Findings 

Delete or upgrade faulty test items and unreliable 
tests. Add extra teaching aids or new materials for 
helping those types of students who were least able 
to succeed with the materials. 
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Stage Nine: Production 

Do this only if you are going to publish and market 
y^ur materials. Edit, mark-up, type-set, format, 
print, and package the materials for formal publi- 
cation. 

Stage Ten:. Publish and Market the Produced Materials 

Establish the price of the publication. Firvalize the 
packaging strategy. Identify, and obtain mailing 
address labels for, ta/rgeted groups of potential 
buyers of the package. Produce and distribute one 
or more flyers to adviertize the package: present or 
demonstrate the package at conventions, conferences, 
book fairs, etc. 

» 

These last rour stages go beyond the scope of this Guide 
because they have nothing especially to do with mastery learn 
ing per se or with the development of educational materials 
per se. They also involve technical matters that require pro 
fessional expert advice, aid, and leadership that may be be- 
yond your organization 1 sj capabilities. For Stages Seven and 
Eight, oytside professional ^valuators shoiild be contracted 
to carry out the field tlpst . Steps Nine and Ten should be 
performed by a publisher \of educational materials. 
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